16 ms·
Maybe you could have a system that heuristicially detects when an crawler is making the request and then feeds them a modified page, itself generated from an LL
by tarpitt 17d ago
Maybe you could have a system that heuristicially detects when an crawler is making the request and then feeds them a modified page, itself generated from an LLM, that injects vulnerabilities and bad code and discussion and such.
- NooneAtAll3 17d agoit's hard to separate spambot that only accesses 3-5 links per IP and a legit user. Changing content for legit user can be devastating
- inigyou 17d agoPut a cookie wall in front. The bot will either load the cookie and have a persistent identifier, or not load the cookie and not get in
- jay_kyburz 17d agowhy is this not the answer? then you can also rate limit each cookie as well.
- sgsjchs 17d agoit'll load the cookie, make one request, move to a different ip, load the cookie, make one request, move to a different ip, ...
- jay_kyburz 17d agoIf you are discovering urls you have to wait for a previous request to finish. The rate limit should work. Requests without a cookie wait 2 seconds. Request with cookies can only make human scale number of requests per second? (1?) If you have a thousands of IP addresses, and you know all the urls you want to request in advance, you can just request them all simultaneously I guess. The next more advanced version is that URLs are unique to your cookie. Users can't share urls anymore, but it might be a tradeoff worth making. Unique urls for each user. You could probably still make this work, if you share your url with another user, they get the page, but heavily rate limited like a regular no cookie request. (a cookie url mismatch gets the rate limited version of the page)
- inigyou 17d agoWhy are you serving new cookies to someone who already has a cookie?
- sgsjchs 17d agoIt discards the cookie after one request and appears as a new user.
- inigyou 16d agoThat implies it is specifically written for your specific website
- sgsjchs 16d agoHow so?
- inigyou 16d agoWhy else would a thing be programmed to drop cookies after every seconds request?
- sgsjchs 16d agoMaybe I misunderstand what you mean by "load the cookie". What is it if not Set-Cookie header with redirect status? That doesn't seem particularly special to me.
- jay_kyburz 16d agoYou don't just redirect, you sleep 2 seconds, then respond with a redirect.
- inigyou 16d agoSet-Cookie header, 200 status and a JavaScript redirect? Many ways. Set-Cookie on pages that load quickly and check it on pages that load slowly?