6 ms·

Upvoted because we’re seeing the same behavior from all AI and Seo bots. They’re BARELY respecting Robots.txt, and hard to block. And when they crawl, they spam

by trebor 2y ago

Upvoted because we’re seeing the same behavior from all AI and Seo bots. They’re BARELY respecting Robots.txt, and hard to block. And when they crawl, they spam and drive up load so high they crash many servers for our clients. If AI crawlers want access they can either behave, or pay. The consequence will almost universal blocks otherwise!

mschuster91 2y ago
Global tarpit is the solution. It makes sense anyway even without taking AI crawlers into account. Back when I had to implement that, I went the semi manual route - parse the access log and any IP address averaging more than X hits a second on /api gets a -j TARPIT with iptables [1]. Not sure how to implement it in the cloud though, never had the need for that there yet. [1] https://gist.github.com/flaviovs/103a0dbf62c67ff371ff75fc62fdded3 https://gist.github.com/flaviovs/103a0dbf62c67ff371ff75fc62f...
- jks 2y ago
  One such tarpit (Nepenthes) was just recently mentioned on Hacker News: https://news.ycombinator.com/item?id=42725147 https://news.ycombinator.com/item?id=42725147 Their site is down at the moment, but luckily they haven't stopped Wayback Machine from crawling it: https://web.archive.org/web/20250117030633/https://zadzmo.org/code/nepenthes/ https://web.archive.org/web/20250117030633/https://zadzmo.or...
  - kazinator 2y ago
    How do you know their site is down? You probably just hit their tarpit. :)
  - marcus0x62 2y ago
    Quixotic[0] (my content obfuscator) includes a tarpit component, but for something like this, I think the main quixotic tool would be better - you run it against your content once, and it generates a pre-obfuscated version of it. It takes a lot less of your resources to serve than dynamically generating the tarpit links and content. 0 - https://marcusb.org/hacks/quixotic.html https://marcusb.org/hacks/quixotic.html
- bwfan123 2y ago
  i would think public outcry by influencers on social media (such as this thread) is a better deterrent, and also establishes a public datapoint and exhibit for future reference.. as it is hard to scale the tarpit.
- idlewords 2y ago
  This doesn't work with the kind of highly distributed crawling that is the problem now.
- seethenerdz 2y ago
  Don't we have intellectual property law for this tho?
gundmc 2y ago
What do you mean by "barely" respecting robots.txt? Wouldn't that be more binary? Are they respecting some directives and ignoring others?
- unsnap_biceps 2y ago
  I believe that a number of AI bots only respect robot.txt entries that explicitly define their static user agent name. They ignore wildcards in user agents. That counts as barely imho. I found this out after OpenAI was decimating my site and ignoring the wildcard deny all. I had to add entires specifically for their three bots to get them to stop.
  - noman-land 2y ago
    This is highly annoying and rude. Is there a complete list of all known bots and crawlers?
    - jsheard 2y ago
      https://darkvisitors.com/agents https://darkvisitors.com/agents https://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt
  - joecool1029 2y ago
    Even some non-profit ignore it now, Internet Archive stopped respecting it years ago: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
    - SR2Z 2y ago
      IA actually has technical and moral reasons to ignore robots.txt. Namely, they want to circumvent this stuff because their goal is to archive EVERYTHING.
      - 2y ago
herpdyderp 2y ago
> The consequence will almost universal blocks otherwise! How? The difficulty of doing that is the problem, isn't it? (Otherwise we'd just be doing that already.)
- ADeerAppeared 2y ago
  > (Otherwise we'd just be doing that already.) Not quite what the original commenter meant but: WE ARE. A major consequence of this reckless AI scraping is that it turbocharged the move away from the web and into closed ecosystems like Discord. Away from the prying eyes of most AI scrapers ... and the search engine indexes that made the internet so useful as an information resource. Lots of old websites & forums are going offline as their hosts either cannot cope with the load or send a sizeable bill to the webmaster who then pulls the plug.
Vampiero 2y ago
> The consequence will almost universal blocks otherwise! Who cares? They've already scraped the content by then.
- jsheard 2y ago
  Bold to assume that an AI scraper won't come back to download everything again, just in case there's any new scraps of data to extract. OP mentioned in the other thread that this bot had pulled 3TB so far, and I doubt their git server actually has 3TB of unique data, so the bot is probably pulling the same data over and over again.
  - xena 2y ago
    FWIW that includes other scrapers, Amazon's is just the one that showed up the most in the logs.
- _heimdall 2y ago
  If they only needed a one-time scrape we really wouldn't be seeing noticeable not traffic today.
- seethenerdz 2y ago
  That's the spirit!
ksec 2y ago
Is there some way website can sell those Data to AI bot in a large zip file rather than being constantly DDoS? Or they could at least have the curtesy to scrap during night time / off peak hours.
- jsheard 2y ago
  No, because they won't pay for anything they can get for free. There's only one situation where an AI company will pay for data, and that's when it's owned by someone with scary enough lawyers to pressure them into paying up. Hence why OpenAI has struck licensing deals with a handful of companies while continuing to bulk-scrape unlicensed data from everyone else.
- tredre3 2y ago
  There is project whose goal is to avoid this crawling-induced DDoS by maintaining a single web index: https://commoncrawl.org/ https://commoncrawl.org/
- seethenerdz 2y ago
  Is existing intellectual property law not sufficient? Why aren't companies being prosecuted for large-scale theft?
emmelaich 2y ago
If they're AI bots it might be fun to feed them nonsense. Just send hack megabytes of "Bezos is a bozo" or something like that. Even more fun if you could cooperate with many other otherwise-unrelated websites, e.g. via time settings in a modified tarpit.
seethenerdz 2y ago
Don't worry, though, because IP law only applies to peons like you and me. :)

by trebor 2y ago

mschuster91 2y ago

Global tarpit is the solution. It makes sense anyway even without taking AI crawlers into account. Back when I had to implement that, I went the semi manual route - parse the access log and any IP address averaging more than X hits a second on /api gets a -j TARPIT with iptables [1]. Not sure how to implement it in the cloud though, never had the need for that there yet. [1] https://gist.github.com/flaviovs/103a0dbf62c67ff371ff75fc62fdded3 https://gist.github.com/flaviovs/103a0dbf62c67ff371ff75fc62f...

jks 2y ago
One such tarpit (Nepenthes) was just recently mentioned on Hacker News: https://news.ycombinator.com/item?id=42725147 https://news.ycombinator.com/item?id=42725147 Their site is down at the moment, but luckily they haven't stopped Wayback Machine from crawling it: https://web.archive.org/web/20250117030633/https://zadzmo.org/code/nepenthes/ https://web.archive.org/web/20250117030633/https://zadzmo.or...
- kazinator 2y ago
  How do you know their site is down? You probably just hit their tarpit. :)
- marcus0x62 2y ago
  Quixotic[0] (my content obfuscator) includes a tarpit component, but for something like this, I think the main quixotic tool would be better - you run it against your content once, and it generates a pre-obfuscated version of it. It takes a lot less of your resources to serve than dynamically generating the tarpit links and content. 0 - https://marcusb.org/hacks/quixotic.html https://marcusb.org/hacks/quixotic.html
bwfan123 2y ago
i would think public outcry by influencers on social media (such as this thread) is a better deterrent, and also establishes a public datapoint and exhibit for future reference.. as it is hard to scale the tarpit.
idlewords 2y ago
This doesn't work with the kind of highly distributed crawling that is the problem now.
seethenerdz 2y ago
Don't we have intellectual property law for this tho?

gundmc 2y ago

What do you mean by "barely" respecting robots.txt? Wouldn't that be more binary? Are they respecting some directives and ignoring others?

unsnap_biceps 2y ago
I believe that a number of AI bots only respect robot.txt entries that explicitly define their static user agent name. They ignore wildcards in user agents. That counts as barely imho. I found this out after OpenAI was decimating my site and ignoring the wildcard deny all. I had to add entires specifically for their three bots to get them to stop.
- noman-land 2y ago
  This is highly annoying and rude. Is there a complete list of all known bots and crawlers?
  - jsheard 2y ago
    https://darkvisitors.com/agents https://darkvisitors.com/agents https://github.com/ai-robots-txt/ai.robots.txt https://github.com/ai-robots-txt/ai.robots.txt
- joecool1029 2y ago
  Even some non-profit ignore it now, Internet Archive stopped respecting it years ago: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engines-dont-work-well-for-web-archives/ https://blog.archive.org/2017/04/17/robots-txt-meant-for-sea...
  - SR2Z 2y ago
    IA actually has technical and moral reasons to ignore robots.txt. Namely, they want to circumvent this stuff because their goal is to archive EVERYTHING.
    - 2y ago

herpdyderp 2y ago

> The consequence will almost universal blocks otherwise! How? The difficulty of doing that is the problem, isn't it? (Otherwise we'd just be doing that already.)

ADeerAppeared 2y ago
> (Otherwise we'd just be doing that already.) Not quite what the original commenter meant but: WE ARE. A major consequence of this reckless AI scraping is that it turbocharged the move away from the web and into closed ecosystems like Discord. Away from the prying eyes of most AI scrapers ... and the search engine indexes that made the internet so useful as an information resource. Lots of old websites & forums are going offline as their hosts either cannot cope with the load or send a sizeable bill to the webmaster who then pulls the plug.

Vampiero 2y ago

> The consequence will almost universal blocks otherwise! Who cares? They've already scraped the content by then.

jsheard 2y ago
Bold to assume that an AI scraper won't come back to download everything again, just in case there's any new scraps of data to extract. OP mentioned in the other thread that this bot had pulled 3TB so far, and I doubt their git server actually has 3TB of unique data, so the bot is probably pulling the same data over and over again.
- xena 2y ago
  FWIW that includes other scrapers, Amazon's is just the one that showed up the most in the logs.
_heimdall 2y ago
If they only needed a one-time scrape we really wouldn't be seeing noticeable not traffic today.
seethenerdz 2y ago
That's the spirit!

ksec 2y ago

Is there some way website can sell those Data to AI bot in a large zip file rather than being constantly DDoS? Or they could at least have the curtesy to scrap during night time / off peak hours.

jsheard 2y ago
No, because they won't pay for anything they can get for free. There's only one situation where an AI company will pay for data, and that's when it's owned by someone with scary enough lawyers to pressure them into paying up. Hence why OpenAI has struck licensing deals with a handful of companies while continuing to bulk-scrape unlicensed data from everyone else.
tredre3 2y ago
There is project whose goal is to avoid this crawling-induced DDoS by maintaining a single web index: https://commoncrawl.org/ https://commoncrawl.org/
seethenerdz 2y ago
Is existing intellectual property law not sufficient? Why aren't companies being prosecuted for large-scale theft?

emmelaich 2y ago

If they're AI bots it might be fun to feed them nonsense. Just send hack megabytes of "Bezos is a bozo" or something like that. Even more fun if you could cooperate with many other otherwise-unrelated websites, e.g. via time settings in a modified tarpit.

seethenerdz 2y ago

Don't worry, though, because IP law only applies to peons like you and me. :)