5 ms·

Agreed; we have a Linode VPS for some small WordPress websites and it was grinding to a halt after an AI crawler started iterating through every possible date i

by rob 2mo ago

Agreed; we have a Linode VPS for some small WordPress websites and it was grinding to a halt after an AI crawler started iterating through every possible date in "The Events Calendar"’s URL structure, sequentially scraping calendar pages all the way back to like 1970. I'm sure that's partially my fault for not putting it behind Cloudflare or anything though.

account42 2mo ago
The response to that should be to restrict the events calendar and similar dynamic functionality, not to restrict the 99% of users who are accessing what should be 100% staticly cached HTML.
- rob 2mo ago
  Good idea; I'll make sure we have all random ?date=1970-01-01, ?date=1970-01-02, all the way to ?date=2026-07-22 URLs cached, even the ones that are 404s but the crawlers still try anyways. Thank you!
  - account42 2mo ago
    It's always entertaining when people proudly display their lack reading comprehension.
    - rob 2mo ago
      That's very true.
- dormento 2mo ago
  Of course, the _actual_ response to that should be to make the AI companies somehow liable for the massive disruption they caused (and still cause). But alas, we can't have nice things.
  - inigyou 2mo ago
    There's no evidence this massive DDoS attack is related to AI.