6 ms·

why is this not the answer? then you can also rate limit each cookie as well.

sgsjchs 19d ago
it'll load the cookie, make one request, move to a different ip, load the cookie, make one request, move to a different ip, ...
- jay_kyburz 19d ago
  If you are discovering urls you have to wait for a previous request to finish. The rate limit should work. Requests without a cookie wait 2 seconds. Request with cookies can only make human scale number of requests per second? (1?) If you have a thousands of IP addresses, and you know all the urls you want to request in advance, you can just request them all simultaneously I guess. The next more advanced version is that URLs are unique to your cookie. Users can't share urls anymore, but it might be a tradeoff worth making. Unique urls for each user. You could probably still make this work, if you share your url with another user, they get the page, but heavily rate limited like a regular no cookie request. (a cookie url mismatch gets the rate limited version of the page)
- inigyou 19d ago
  Why are you serving new cookies to someone who already has a cookie?
  - sgsjchs 19d ago
    It discards the cookie after one request and appears as a new user.
    - inigyou 19d ago
      That implies it is specifically written for your specific website
      - sgsjchs 18d ago
        How so?
        inigyou 18d ago
        Why else would a thing be programmed to drop cookies after every seconds request?
        sgsjchs 18d ago
        Maybe I misunderstand what you mean by "load the cookie". What is it if not Set-Cookie header with redirect status? That doesn't seem particularly special to me.
        jay_kyburz 18d ago
        You don't just redirect, you sleep 2 seconds, then respond with a redirect.
        inigyou 18d ago
        Set-Cookie header, 200 status and a JavaScript redirect? Many ways. Set-Cookie on pages that load quickly and check it on pages that load slowly?
        sgsjchs 17d ago
        What happens when a user without the cookie GETs a slow page? Does he get redirected to fast one to and back?

it'll load the cookie, make one request, move to a different ip, load the cookie, make one request, move to a different ip, ...

jay_kyburz 19d ago
If you are discovering urls you have to wait for a previous request to finish. The rate limit should work. Requests without a cookie wait 2 seconds. Request with cookies can only make human scale number of requests per second? (1?) If you have a thousands of IP addresses, and you know all the urls you want to request in advance, you can just request them all simultaneously I guess. The next more advanced version is that URLs are unique to your cookie. Users can't share urls anymore, but it might be a tradeoff worth making. Unique urls for each user. You could probably still make this work, if you share your url with another user, they get the page, but heavily rate limited like a regular no cookie request. (a cookie url mismatch gets the rate limited version of the page)
inigyou 19d ago
Why are you serving new cookies to someone who already has a cookie?
- sgsjchs 19d ago
  It discards the cookie after one request and appears as a new user.
  - inigyou 19d ago
    That implies it is specifically written for your specific website
    - sgsjchs 18d ago
      How so?
      - inigyou 18d ago
        Why else would a thing be programmed to drop cookies after every seconds request?
        sgsjchs 18d ago
        Maybe I misunderstand what you mean by "load the cookie". What is it if not Set-Cookie header with redirect status? That doesn't seem particularly special to me.
        jay_kyburz 18d ago
        You don't just redirect, you sleep 2 seconds, then respond with a redirect.
        inigyou 18d ago
        Set-Cookie header, 200 status and a JavaScript redirect? Many ways. Set-Cookie on pages that load quickly and check it on pages that load slowly?
        sgsjchs 17d ago
        What happens when a user without the cookie GETs a slow page? Does he get redirected to fast one to and back?