6 ms·

But what do these bots gain from this?

• esseph 1mo ago
  The most recent info of every single fucking thing on the web. Everything. Data locusts.
• bredren 1mo ago
  Possibly content that appears only briefly? I'm not sure otherwise. It seems wasteful.
• ddxv 1mo ago
  It's all OpenAI / Anthropic / Singapore/China based crawlers. I guess they gain data that I provide for free. I get very little crawling by Googlebot / Bing etc which barely even index my site, I only have a single page "indexed" by Google.
• AlienRobot 1mo ago
  Have you ever use an AI chatbot? It's basically a democratized scraper. Type something stupid, it "visits" 200 websites and regurgitates some random gibberish.