braden_e

5 ms·

1. 1.▲
   by braden_e 1y ago

   There is a very large scale crawler that uses random valid user agents and a staggeringly large pool of ips. I first noticed it because a lot of traffic was coming from Brazil and "HostRoyale" (asn 203020). They send only a few re
2. 2.▲
   by braden_e 2y ago

   ClaudeBot (second worst crawler) and GPTBot are the only ones that identify themselves in an obvious way. The rest I am blocking by network. I assume AI when the crawler is very aggressively downloading images - it must be costing them an a
3. 3.▲
   by braden_e 2y ago

   This is the worst behaved bot I have ever seen, I suspect it is AI related. I recently decided to block all the AI crawlers - unlike search engines I get nothing from them.
4. 4.▲
   by braden_e 15y ago

   Other than microsoft themselves: myspace is still fairly huge but probably a mixed stack. Plentyoffish is up there too - http://highscalability.com/plentyoffish-architecture