5 ms·

Not sure the emotive language is warranted. Message appears to be “if you use robots.txt AND archive sites honor it AND you are dumb enough to delete your data

Not sure the emotive language is warranted. Message appears to be “if you use robots.txt AND archive sites honor it AND you are dumb enough to delete your data without a backup THEN you won’t have a way to recover and you’ll be sorry”. It also presumes that dealing with automated traffic is a solved problem, which with the volumes of LLM scraping going on, is simply not true for more hobbyist setups.

bigbuppo 1y ago
Or major web properties for that matter.
paulddraper 1y ago
> volumes of LLM scraping FWIW I have not seen a reputable report on the % of web scraping in the past 3 years. (Wikipedia being a notable exception...but I would guess Wikipedia to see a far larger increase than anything else.)
- esseph 1y ago
  It's hard because of attribution, but it absolutely is happening at very high volume. I actually got an alert this morning when I woke up from our monitoring tools that some external sites were being scraped. Happens multiple times a day. A lot of it is coming through compromised residential endpoint botnets.
  - paulddraper 1y ago
    Even without attribution…seeing bot traffic or general traffic increase
- tolmasky 1y ago
  Wikipedia says their traffic increased roughly 50% [1] from AI bots, which is a lot, sure, but nowhere near the amount where you'd have to rearchitect your site or something. And this checks out, if it was actually debilitating, you'd notice Wikipedia's performance degrade. It hasn't. You'd see them taking some additional steps to combat this. They haven't. Their CDN handles it just fine. They don't even both telling AI bots to just download the tarballs they specifically make available for this exact use case. More importantly, Wikipedia almost certainly represents the ceiling of traffic increase. But luckily, we don't have to work with such coarse estimation, because according to Cloudflare, the total increase from combined search and AI bots in the last year (May 2024 - May 2025), has just been... 18% [2]. The way you hear people talk about it though, you'd think that servers are now receiving DDOS-levels of traffic or something. For the life of me I have not been able to find a single verifiable case of this. Which if you think about it makes sense... It's hard to generate that sort of traffic, that's one of the reasons people pay for botnets. You don't bring a site to its knees merely by accidentally "not making your scraper efficient". So the only other possible explanation would be such a larger number of scrapers simultaneously but independently hitting sites. But this also doesn't check out. There aren't thousands of different AI scrapers out there that in aggregate are resulting in huge traffic spikes [2]. Again, the total combined increase is 18%. The more you look into this accepted idea that we are in some sort of AI scraping traffic apocalypse, the less anything makes sense. You then look at this Anubis "AI scraping mitigator" and... I dunno. The author contends that one if its tricks is that it not only uses JavaScript, but "modern JavaScript like ES6 modules," and that this is one of the ways it detects/prevents AI scrapers [3]. No one is rolling their own JS engine for a scraper such that they are being blocked from their inability to keep up with the latest ECMAScript spec. You are just using an existing JS engine, all of which support all these features. It would actually be a challenge to find an old JS engine these days. The entire things seems to be built on the misconception that the "common" way to build a scraper is doing something curl-esque. This idea is entirely based on the google scraper which itself doesn't even work that way anymore, and only ever did because it was written in the 90s. Everyone that rolls their own scraper these days just uses Puppeteer. It is completely unrealistic to make a scraper that doesn't run JavaScript and wait for the page to "settle down" because so many pages, even blogs, are just entirely client-side rendered SPAs. If I were to write a quick and dirty scraper today I would trivially make it through Anubis' protections... by doing literally nothing and without even realizing Anubis exists. Just using standard scraping practices with Puppeteer. Meanwhile Anubis is absolutely blocking plenty of real humans, with the author for example telling people to turn on cookies so that Anubis can do its job [4]. I don't think Anubis is blocking anything other than humans and Message's link preview generator. I'm investigating further, but I think this entire thing may have started due to some confusion, but want to see if I can actually confirm this before speculating further. 1. https://www.techspot.com/news/107407-wikipedia-servers-struggling-under-pressure-ai-scraping-bots.html https://www.techspot.com/news/107407-wikipedia-servers-strug... (notice the clickbait title vs. the actual contents) 2. https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/?utm_source=chatgpt.com/#general-ai-and-search-crawling-growth-18 https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-cr... 3. https://codeberg.org/forgejo/discussions/issues/319#issuecomment-3486873 https://codeberg.org/forgejo/discussions/issues/319#issuecom... 4. https://github.com/TecharoHQ/anubis/issues/964#issuecomment-3172689053 https://github.com/TecharoHQ/anubis/issues/964#issuecomment-...
  - xena 1y ago
    Hi, main author of Anubis here. How am I meant to store state like "user passed a check" without cookies? Please advise.
    - tolmasky 1y ago
      If the rest of my post is accurate, that's not the actual concern, right? Since I'm not sure if the check itself is meaningful. From what is described in the documentation [1], I think the practical effect of this system is to block users running old mobile browsers or running browsers like Opera Mini in third world countries where data usage is still prohibitively expensive. Again, the off-the-shelf scraping tools [2] will be unaffected by any of this, since they're all built on top of Puppeteer, and additionally are designed to deal with the modern SPA web which is (depressingly) more or less isomorphic to a "proof-of-work". If you are open to jumping on a call in the next week or two I'd love to discuss directly. Without going into a ton of detail, I originally started looking into this because the group I'm working with is exploring potentially funding a free CDN service for open source projects. Then this AI scraper stuff started popping up, and all of a sudden it looked like if these reports were true it might make such a project no longer economically realistic. So we started trying to collect data and concretely nail down what we'd be dealing with and what this "post-AI" traffic looks like. As such, I think we're 100% aligned on our goals. I'm just trying to understand what's going on here since none of the second-order effects you'd expect from this sort of phenomenon seem to be present, and none of the places where we actually have direct data seem to show this taking place (and again, Cloudflare's data seems to also agree with this). But unless you already own a CDN, it's very hard to get a good sense of what's going on globally. So I am totally willing to believe this is happening, and am very incentivized to help if so. EDIT: My email is my HN username at gmail.com if you want to schedule something. 1. https://anubis.techaro.lol/docs/design/how-anubis-works https://anubis.techaro.lol/docs/design/how-anubis-works 2. https://apify.com/apify/puppeteer-scraper https://apify.com/apify/puppeteer-scraper
    - rafram 1y ago
      Cloudflare Turnstile doesn't require cookies. It stores per-request "user passed a check" state using a query parameter. So disabling cookies will just cause you to get a challenge on every request, which is annoying but ultimately fair IMO.
QuercusMax 1y ago
I just plain don't understand what they mean by "suicide note" in this case, and it doesn't seem to be explained in the text. A better analogy would be "Robots.txt is a note saying your backdoor might be unlocked".
- chao- 1y ago
  I also cannot figure out from context what part of this is "suicide". I don't even think it's a note saying your back door is unlocked? As myself and others shared in a sibling comment thread, we have worked at places that implemented robots.txt in order to prevent bots from getting into nearly-infinite tarpits of links that lead to nearly-identical pages.
- stickfigure 1y ago
  The meaning is reasonably clear to me: Robots.txt says "Don't archive this data. When the website dies, all the information dies with it." It's a kind of death pact.
  - QuercusMax 1y ago
    That's not a suicide note, though, in any way I understand it.
    - stickfigure 1y ago
      It's the inevitable suicide of the data. Language gets weird when you anthropomorphize abstract things like "data", but I thought it was clever enough. YMMV.
      - QuercusMax 1y ago
        The suicide of the data listed in robots.txt? How? The whole point of the article is they ignore what you have written in your robots.txt, so they'll archive it regardless of what you say.
        1y ago

bigbuppo 1y ago

Or major web properties for that matter.

paulddraper 1y ago

> volumes of LLM scraping FWIW I have not seen a reputable report on the % of web scraping in the past 3 years. (Wikipedia being a notable exception...but I would guess Wikipedia to see a far larger increase than anything else.)

esseph 1y ago
It's hard because of attribution, but it absolutely is happening at very high volume. I actually got an alert this morning when I woke up from our monitoring tools that some external sites were being scraped. Happens multiple times a day. A lot of it is coming through compromised residential endpoint botnets.
- paulddraper 1y ago
  Even without attribution…seeing bot traffic or general traffic increase
tolmasky 1y ago
Wikipedia says their traffic increased roughly 50% [1] from AI bots, which is a lot, sure, but nowhere near the amount where you'd have to rearchitect your site or something. And this checks out, if it was actually debilitating, you'd notice Wikipedia's performance degrade. It hasn't. You'd see them taking some additional steps to combat this. They haven't. Their CDN handles it just fine. They don't even both telling AI bots to just download the tarballs they specifically make available for this exact use case. More importantly, Wikipedia almost certainly represents the ceiling of traffic increase. But luckily, we don't have to work with such coarse estimation, because according to Cloudflare, the total increase from combined search and AI bots in the last year (May 2024 - May 2025), has just been... 18% [2]. The way you hear people talk about it though, you'd think that servers are now receiving DDOS-levels of traffic or something. For the life of me I have not been able to find a single verifiable case of this. Which if you think about it makes sense... It's hard to generate that sort of traffic, that's one of the reasons people pay for botnets. You don't bring a site to its knees merely by accidentally "not making your scraper efficient". So the only other possible explanation would be such a larger number of scrapers simultaneously but independently hitting sites. But this also doesn't check out. There aren't thousands of different AI scrapers out there that in aggregate are resulting in huge traffic spikes [2]. Again, the total combined increase is 18%. The more you look into this accepted idea that we are in some sort of AI scraping traffic apocalypse, the less anything makes sense. You then look at this Anubis "AI scraping mitigator" and... I dunno. The author contends that one if its tricks is that it not only uses JavaScript, but "modern JavaScript like ES6 modules," and that this is one of the ways it detects/prevents AI scrapers [3]. No one is rolling their own JS engine for a scraper such that they are being blocked from their inability to keep up with the latest ECMAScript spec. You are just using an existing JS engine, all of which support all these features. It would actually be a challenge to find an old JS engine these days. The entire things seems to be built on the misconception that the "common" way to build a scraper is doing something curl-esque. This idea is entirely based on the google scraper which itself doesn't even work that way anymore, and only ever did because it was written in the 90s. Everyone that rolls their own scraper these days just uses Puppeteer. It is completely unrealistic to make a scraper that doesn't run JavaScript and wait for the page to "settle down" because so many pages, even blogs, are just entirely client-side rendered SPAs. If I were to write a quick and dirty scraper today I would trivially make it through Anubis' protections... by doing literally nothing and without even realizing Anubis exists. Just using standard scraping practices with Puppeteer. Meanwhile Anubis is absolutely blocking plenty of real humans, with the author for example telling people to turn on cookies so that Anubis can do its job [4]. I don't think Anubis is blocking anything other than humans and Message's link preview generator. I'm investigating further, but I think this entire thing may have started due to some confusion, but want to see if I can actually confirm this before speculating further. 1. https://www.techspot.com/news/107407-wikipedia-servers-struggling-under-pressure-ai-scraping-bots.html https://www.techspot.com/news/107407-wikipedia-servers-strug... (notice the clickbait title vs. the actual contents) 2. https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-crawling-your-site-in-2025/?utm_source=chatgpt.com/#general-ai-and-search-crawling-growth-18 https://blog.cloudflare.com/from-googlebot-to-gptbot-whos-cr... 3. https://codeberg.org/forgejo/discussions/issues/319#issuecomment-3486873 https://codeberg.org/forgejo/discussions/issues/319#issuecom... 4. https://github.com/TecharoHQ/anubis/issues/964#issuecomment-3172689053 https://github.com/TecharoHQ/anubis/issues/964#issuecomment-...
- xena 1y ago
  Hi, main author of Anubis here. How am I meant to store state like "user passed a check" without cookies? Please advise.
  - tolmasky 1y ago
    If the rest of my post is accurate, that's not the actual concern, right? Since I'm not sure if the check itself is meaningful. From what is described in the documentation [1], I think the practical effect of this system is to block users running old mobile browsers or running browsers like Opera Mini in third world countries where data usage is still prohibitively expensive. Again, the off-the-shelf scraping tools [2] will be unaffected by any of this, since they're all built on top of Puppeteer, and additionally are designed to deal with the modern SPA web which is (depressingly) more or less isomorphic to a "proof-of-work". If you are open to jumping on a call in the next week or two I'd love to discuss directly. Without going into a ton of detail, I originally started looking into this because the group I'm working with is exploring potentially funding a free CDN service for open source projects. Then this AI scraper stuff started popping up, and all of a sudden it looked like if these reports were true it might make such a project no longer economically realistic. So we started trying to collect data and concretely nail down what we'd be dealing with and what this "post-AI" traffic looks like. As such, I think we're 100% aligned on our goals. I'm just trying to understand what's going on here since none of the second-order effects you'd expect from this sort of phenomenon seem to be present, and none of the places where we actually have direct data seem to show this taking place (and again, Cloudflare's data seems to also agree with this). But unless you already own a CDN, it's very hard to get a good sense of what's going on globally. So I am totally willing to believe this is happening, and am very incentivized to help if so. EDIT: My email is my HN username at gmail.com if you want to schedule something. 1. https://anubis.techaro.lol/docs/design/how-anubis-works https://anubis.techaro.lol/docs/design/how-anubis-works 2. https://apify.com/apify/puppeteer-scraper https://apify.com/apify/puppeteer-scraper
  - rafram 1y ago
    Cloudflare Turnstile doesn't require cookies. It stores per-request "user passed a check" state using a query parameter. So disabling cookies will just cause you to get a challenge on every request, which is annoying but ultimately fair IMO.

QuercusMax 1y ago

I just plain don't understand what they mean by "suicide note" in this case, and it doesn't seem to be explained in the text. A better analogy would be "Robots.txt is a note saying your backdoor might be unlocked".

chao- 1y ago
I also cannot figure out from context what part of this is "suicide". I don't even think it's a note saying your back door is unlocked? As myself and others shared in a sibling comment thread, we have worked at places that implemented robots.txt in order to prevent bots from getting into nearly-infinite tarpits of links that lead to nearly-identical pages.
stickfigure 1y ago
The meaning is reasonably clear to me: Robots.txt says "Don't archive this data. When the website dies, all the information dies with it." It's a kind of death pact.
- QuercusMax 1y ago
  That's not a suicide note, though, in any way I understand it.
  - stickfigure 1y ago
    It's the inevitable suicide of the data. Language gets weird when you anthropomorphize abstract things like "data", but I thought it was clever enough. YMMV.
    - QuercusMax 1y ago
      The suicide of the data listed in robots.txt? How? The whole point of the article is they ignore what you have written in your robots.txt, so they'll archive it regardless of what you say.
      - 1y ago