morusrubra

5 ms·

1. 1.▲
   Mitigating Aggressive Crawler Traffic in the Age of Generative AI(journal.code4lib.org)

   2 points by morusrubra 10mo ago|0 comments
2. 2.▲
   by morusrubra 1y ago

   Although I think it's likely that these are "AI" bots, the real problem is the proliferation of rich and crappy crawlers. Whether or not legacy crawlers respect robots.txt, etc., they do seem to be sophisticated enough to det