10 ms·

Cloudflare Introduces Default Blocking of A.I. Data Scrapers

429 points by stephendause 1y ago

YPPH 1y ago
This is great. But my concerns about Cloudflare's power remain. Today it's blocking AI crawlers, tomorrow will it be blocking all browsers that fail hardware-attestation checks?
cmg 1y ago
Archive link: https://archive.ph/ARnyu https://archive.ph/ARnyu
- joshdavham 1y ago
  How did you make that link?
badlibrarian 1y ago
Did they ever fix the auto-blocking of RSS feeds? https://news.ycombinator.com/item?id=41864632 https://news.ycombinator.com/item?id=41864632
blakesterz 1y ago
The list of bots is pretty short right now: https://developers.cloudflare.com/bots/concepts/bot/#ai-bots https://developers.cloudflare.com/bots/concepts/bot/#ai-bots
- ZiiS 1y ago
  Enough to more than half the traffic to most sites if the blocks hold.
- hennell 1y ago
  Cloudflare sees a lot of the web traffic. I assume these are the biggest bots they're seeing right now, and any new contenders would be added as they find them. Probably impossible to really block everything, but they've got the web-coverage to detect more than most.
  - TechDebtDevin 1y ago
    They are lying. They cant detect crawlers unless we tell them we are who we are.
- JimDabell 1y ago
  > AI bots > You can opt into a managed rule that will block bots that we categorize as artificial intelligence (AI) crawlers (“AI Bots”) from visiting your website. Customers may choose to do this to prevent AI-related usage of their content, such as training large language models (LLM). > CCBot (Common Crawl) Common Crawl is not an AI bot: https://commoncrawl.org https://commoncrawl.org
  - johneth 1y ago
    The data it collects is used by AI companies, though.
Spivak 1y ago
Poor ChatGPT-User, nobody understands you. Blocking a real user because of the, admittedly odd, browser they're using misses the point.
Roark66 1y ago
This is a bit silly. Slowing down, yes, but blocking? People who *really* want that content will find a way and this will hit everyone else instead that will have to do silly riddles before following every link or run crypto mining for them before being shown the content . I recently went to a big local auction site on which I buy frequently and I got one of these "we detected unusual traffic from your network" messages. And "prove you're human". Which was followed by "you completed the capcha in 0.4s your IP is banned". Really? Am I supposed to slow down my browsing now? I tried a different browser, a different OS, logging on,clearing cookies, etc. Same result when I tried a search. It took 4h after contacting their customer service to unblock it. And the explanation was "you're clicking too fast". At some point it just becomes a farce and the hassle is not worth the content. Also, while my story doesn't involve any bots perhaps a time will come when local LLMs will be good enough that I'll be able to tell one "reorder my cat food" and it will go and do it. Why are they so determined to "stop it" (spoiler, they can't). For anyone who says LLMs are already capable of ordering cat food I say not so fast. First the cat food has to be on sale/offer (sometimes combined with extras). Second it is supposed to be healthy (no grains) and third the taste needs to be to my cats liking. So far I'm not going to trust a LLM with this.
- picohernandez 1y ago
  I was chatting with my sister last weekend. As a hobby, she creates and sells wedding invitation and other designs at an online marketplace site called Zazzle. She was telling me all about how that site implemented some automatic bot detection and it sounded like it was a total disaster. Real content creators were getting wrongly flagged as bots and then getting blocked from using the site just for using the most fundamental site functionality, and to make it worse, then it was apparently impossible for them to get past the captcha challenge or whatever it showed next. She forwarded me a link to some support forum discussion about it and it was mindboggling the troubles that some of the content creators there had to go through: https://community.zazzle.com/t5/technical-issues/bot-test-won-t-let-me-into-my-account-press-and-hold-didn-t-work/m-p/205396 https://community.zazzle.com/t5/technical-issues/bot-test-wo... My sister said that her sales figures are way down compared to what they used to be and she didn't know if this bot flagger was disrupting real paying customers too. She said it had flagged her a couple of times, although she was luckily able to get past the bot challenge. She has pretty much given up on making and uploading new designs because of what was happening to other content creators there. She's now scared to use the site because she doesn't want to get wrongly locked out of her account.
Sol- 1y ago
Do the major AI companies actually honor robots.txt? Even if some of their publicly known crawlers might do it, surely they have surreptitious campaigns where they do some hidden crawling, just like how they illegally pirate books, images and user data to train on.
- px43 1y ago
  There's a lack of clarity, but it seems likely to me that a majority of this traffic is actually people asking questions to the AI, and the AI going out and researching for answers. When the AI tools are being used like a web browser to do research, should they still be adhering to robots.txt, or is that only intended for search indexing?
- chasd00 1y ago
  My thought too, honoring robots.txt is just a convention. There's no requirement to follow robots.txt, or at least certainly no technical requirement. I don't think there's any automatic legal requirement either. Maybe sites could add "you must honor policies set in robots.txt" to something like a terms of service but I have no idea if that would have enough teeth for a crawler to give up.
  - TechDebtDevin 1y ago
    Cloudflare snd their customera have been desperately for years trying to kill scrapers in court. This is all. Meaningless, but they are probably gearing up for another legal battle to define robots.txt as a legal contract. Theyre going to use this marketplace theyre scamming people with to do it. They will fail.
  - prmoustache 1y ago
    I don't think terms of service are applicable anyway. Terms of Service aren't a signed contract as you may never see it nor know there is one. This happens both in the case of visiting the site interactively or fetching a page programatically.
- mschuster91 1y ago
  Cloudflare, for all I hate their role as a gatekeeper these days, actually has the leverage to force the AI companies to bend.
btown 1y ago
The headline is somewhat misleading: sites using Cloudflare now have an opt-in option to quickly block all AI bots, but it won't be turned on by default for sites using Cloudflare. The idea that Cloudflare could do the latter at the sole discretion of its leadership, though, is indicative of the level of power Cloudflare holds.
- bitpush 1y ago
  It is now an adversarial relationship between aibots and website, and cloudflare is merely reacting to it. Would you say the same for ddos protection? Isn't that the same as well?
  - TechDebtDevin 1y ago
    They arent doing anything. They are attempting to insert themselves into the middle of a marketplace (that doesnt exist and never will) where scrapers pay for IP. They think theyre going to profit off the bots, not protect your site. Dont fall for their scam.
    - bitpush 1y ago
      What do you mean they are trying to insert themselves? If I have a website that I host with cloudflare, I (as the rightful website owner) has inserted Cloudflare in between. It isnt CF going around saying, that's a nice website you have there. I'm gonna put myself in between.
      - deleted 1y ago
        [deleted]
- GrayShade 1y ago
  > sites using Cloudflare now have an opt-in option to quickly block all AI bots, but it won't be turned on by default for sites using Cloudflare Do you have a source for that? https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/ https://blog.cloudflare.com/content-independence-day-no-ai-c... does say "changing the default".
postalcoder 1y ago
> When you enable this feature via a pre-configured managed rule, Cloudflare can detect and block verified AI bots that comply with robots.txt and respect crawl rates, and do not hide their behavior from your website. The rule has also been expanded to include more signatures of AI bots that do not follow the rules. We already know companies like Perplexity are masking their traffic. I'm sure there's more than meets the eye, but taking this at face value, doesn't punishing respectful and transparent bots only incentivize obfuscation? edit: This link[0], posted in a comment elsewhere, addresses this question. tldr, obfuscation doesn't work. > We leverage Cloudflare global signals to calculate our Bot Score, which for AI bots like the one above, reflects that we correctly identify and score them as a “likely bot.” > When bad actors attempt to crawl websites at scale, they generally use tools and frameworks that we are able to fingerprint. For every fingerprint we see, we use Cloudflare’s network, which sees over 57 million requests per second on average, to understand how much we should trust this fingerprint. To power our models, we compute global aggregates across many signals. Based on these signals, our models were able to appropriately flag traffic from evasive AI bots, like the example mentioned above, as bots. [0] https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo...
- colechristensen 1y ago
  >doesn't punishing respectful and transparent bots only incentivize obfuscation? They're cloudflare and it's not like it's particularly easy to hide a bot that is scraping large chunks of the Internet from them. On top of the fact that they can fingerprint any of your sneaky usage, large companies have to work with them so I can only assume there are channels of communication where cloudflare can have a little talk with you about your bad behavior. I don't know how often lawyers are involved but I would expect them to be.
- jerf 1y ago
  "doesn't punishing respectful and transparent bots only incentivize obfuscation?" Sure, but we crossed that bridge over 20 years ago. It's not creating an arms race where there wasn't already one. Which is my generic response to everyone bringing similar ideas up. "But the bots could just...", yeah, they've been doing it for 20+ years and people have been fighting it for just as long. Not a new problem, not a new set of solutions, no prospect of the arms race ending any time soon, none of this is new.
dougb5 1y ago
> Cloudflare can detect and block verified AI bots that comply with robots.txt and respect crawl rates, and do not hide their behavior from your website It's the bots that do hide their behavior -- via residential proxy services -- that are causing most of the burden, for my site anyway. Not these large commercial AI vendors.
alganet 1y ago
> If A.I. companies freely use data from various websites without permission or payment, people will be discouraged from creating new digital content I don't see a way out of this happening. AI fundamentally discourages other forms of digital interaction as it grows. Its mechanism of growing is killing other kinds of digital content. It will eventually kill the web, which is, ironically, its main source of food.
- preachermon 1y ago
  [flagged]
  - alganet 1y ago
    These kinds of comparisons rarely lead to good discussions. Let's instead be focused and talk about real stuff. Consider https://learnpythonthehardway.org/ https://learnpythonthehardway.org/ for example. It has influenced a generation of Python developers. Not just the main website, but the tons of Python code and Python-related content it inspired. Why would anyone write these kinds of textbooks/websites/guides if AI can replace them? AI companies are effectively broadcasting you don't need the hard way anymore, you can just vibe. Arguibly though, without the existance of Learn Python the Hard Way and similar content, AI would be worse at writing Python stuff. That's what I mean by "main source of food", good content that influences a lot of people. Net-positive effects hard to predict or even identify except for the more popular cases (such as LPTHW). If my prediction is right, no one will notice that good content has stopped being produced. It will appear as if content is being created in generally the same way as before, but in reality, these long tail initiatives like LPTHW will have ceased before anyone can do anything about it. Again, I don't see a way out of this scenario. Not for AI companies, not for content writers. This is going to happen. The world in which I'm wrong is the best one.
    - mfost 1y ago
      In a similar vein, I remember people advocating for replacing new untrained hires with AI. After all, a competent senior engineer is needed to validate the contributions of the new hires anyway and they can do the same checking the AI code. But then, how would you even train and replace those competent seniior engineers that do the filtering when they retire? The whole system was predicated on having a chain of new hires that gain experience in the process.
jasonthorsness 1y ago
I turned this on and it adjusts the robots.txt automatically; not sure what else it is doing. # NOTICE: The collection of content and other data on this # site through automated means, including any device, tool, # or process designed to data mine or scrape content, is # prohibited except (1) for the purpose of search engine indexing or # artificial intelligence retrieval augmented generation or (2) with express # written permission from this site’s operator. # To request permission to license our intellectual # property and/or other materials, please contact this # site’s operator directly. # BEGIN Cloudflare Managed content User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: meta-externalagent Disallow: / # END Cloudflare Managed Content User-agent: * Disallow: /* Allow: /$
- xyst 1y ago
  So in addition to updating the robots.txt file, which really only blocks a small number of them. Seems CF has been gathering data and profiling these malicious agents. This post by CF elaborates a bit further: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo... Basically becomes a game of cat and mouse.
- postalcoder 1y ago
  This is interesting. The reasoning and response don't line up. > Cloudflare is making the change to protect original content on the internet, Mr. Prince said. If A.I. companies freely use data from various websites without permission or payment, people will be discouraged from creating new digital content, he said > prohibited except for the purpose of [..] artificial intelligence retrieval augmented generation This seems to be targeted at taxing training of language models, but why an exclusion for the RAG stuff? That seems like it has a much greater immediate impact for online content creators, for whom the bots are obviating a click.
  - fennecfoxy 1y ago
cratermoon 1y ago
I'm still not sure this is going to be very effective, as so many of the worst offenders don't identify themselves as bots, and often change their user agent. Has Cloudflare said anything about identifying the bad actors?
- chasd00 1y ago
  i've mentioned this in a couple replies so maybe i'm wrong but it's up to the client to obey robots.txt. Why would they not just ignore it? Unless there's some legal consequence not complying with robots.txt then why even follow it? There's no technical enforcement of the policies in the file, it's up to the client to honor them.
  - kentonv 1y ago
    > There's no technical enforcement of the policies in the file, it's up to the client to honor them. That's incorrect. Cloudflare does in fact enforce this at a technical level. Cloudflare has been doing bot detection for years and can pretty reliably detect when bots are not following robots.txt and then block them.
- GrayShade 1y ago
  Yes, they have over the years, for example https://blog.cloudflare.com/residential-proxy-bot-detection-using-machine-learning/ https://blog.cloudflare.com/residential-proxy-bot-detection-..., https://blog.cloudflare.com/cloudflare-bot-management-machine-learning-and-more/ https://blog.cloudflare.com/cloudflare-bot-management-machin..., https://blog.cloudflare.com/introducing-bot-analytics/ https://blog.cloudflare.com/introducing-bot-analytics/.
  - cratermoon 1y ago
    Between those measures, if they are effective and the new blocking, maybe the bigger companies will be induced to behave a little better.
lucasyvas 1y ago
I fail to see how this won’t just result in UA string or other obfuscation.
- chasd00 1y ago
  a crawler doesn't have to change anything, they can just ignore the robots.txt file. It's up to the client to read robots.txt and follow its directives but there's no technical reason why the client cannot just ignore everything in the file period.
- kube-system 1y ago
  Cloudflare’s filtering is already way more sophisticated than just looking at UA string or other voluntary reporting. They’re almost certainly using fingerprinting and behavioral analytics.
gazpacho 1y ago
From an open source projects perspective we’d want to disable this on our docs sites. We actually want those to be very discoverable by LLMs, during training or online usage.
rorylaitila 1y ago
Unfortunately I think pissing into the wind. Information websites are all but dead. AI contains all published human information. If you have positioned your website as an answer to a question, it won't survive that way. "Information" is dead but content is not. Stories, empathy, community, connection, products, services. Content of this variety is exploding. The big challenge is discoverability. Before, information arbitrage was one pathway to get your content discovered, or to skim a profit. This is over with AI. New means of discovery are necessary, largely network and community based. AI will throw you a few bones, but it will be 10% of what SEO did.
- fennecfoxy 1y ago
  >AI contains all published human information No, it most certainly does not. It was certainly trained on large swathes of human knowledge/interactions. A model that consists of a perfect representation/compression of all this info is a zip file, not a model file.
  - rorylaitila 1y ago
    AI providers have scrapped and will continue to, all internet published information or virtually so. Since "Information" is infinite, AI cannot contain "all information" in a complete sense. But it certainly answers almost everything that matters for any existing search query that has ever been targeted by a webpage that is crawlable. In any case, as manifest by real world SEO, which is plummeting in traffic for informational queries, the effect is the same. This real world impact is what matters and will not be reversed, regardless of attempts at blocking.
- ozgrakkurt 1y ago
  You are assuming LLMs will replace search engines. Why is this the case? To me it seems like there has to be so much optimization for this to happen that, it is not likely. LLM answers are slow and unreliable. Even using something like perplexity doesn’t give much value over using a regular search engine in my experience
  - 1y ago
Meekro 1y ago
I've heard lots of people on HN complaining about bot traffic bogging down their websites, and as a website operator myself I'm honestly puzzled. If you're already using Cloudflare, some basic cache configuration should guarantee that most bot traffic hits the cache and doesn't bog down your servers. And even if you don't want to do that, bandwidth and CPU are so cheap these days that it shouldn't make a difference. Why is everyone so upset?
- deepsiml 1y ago
  Not much into that kind of DevOps. What is a good basic caching in this instance?
  - TechDebtDevin 1y ago
    Cloudflare and other CDNs will usually automatically cache your static pages.
  - haiku2077 1y ago
    It comes down to: 1. Use the Cache-Control header to express how to cache your site correctly (https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Caching https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Cac...) 2. Use a CDN service, or at least a caching reverse proxy, to serve most of the cacheable requests to reduce load on the (typically much more expensive) origin servers
    - mrweasel 1y ago
      Just note that many AI scrapers will go to great length to do cache busting. For some reason many of them feel like they need to get the absolute latest version and don't trust your cache.
      - haiku2077 1y ago
        You can use Cache Control headers to express that your own CDN should aggressively refresh a resource but always serve it to external clients from cache. It's covered in the link under "Managed Caches"
deleted 1y ago
[deleted]
yodon 1y ago
Discussed yesterday (270+ comments)[0] [0]https://news.ycombinator.com/item?id=44432385 https://news.ycombinator.com/item?id=44432385
dawnerd 1y ago
I’ve been using this for a while on my mastodon server and after a few tweaks to make sure it wasn’t blocking legit traffic it’s been really working great. Between Microsoft and Meta, they were hitting my services more than any other traffic combined which says a lot of you know how noisy mastodon can be. Server load went down dramatically. It also completely put a stop to perplexity as far as I can tell. And the robots file meant nothing, they’d still request it hundreds of thousands of times instead of caching it. Every request they’d hit it first then hit their intended url.
- TechDebtDevin 1y ago
  This does nothing dude. Literally nothing. OpenAI or whoever are just going to hire people like me who dont get caught. Stop ruining the experience of users and allowing cf to fill the internet with more bloated javascript challenge pages and privacy invading fingerprinting. Stop making cf the police of the internet. We're literally handing the internet to this company on a silver platter to do MITM attacks on our privacy and god knows what else. Fucking wild.
  - fluidcruft 1y ago
    They literally said it significantly reduced their server resource usage. Are you suggesting they are lying?
  - drowsspa 1y ago
    Why do you think you have the moral high ground here?
  - dawnerd 1y ago
    Well the alternative is to not have an instance at all so… what do you suggest? I’m not paying for the other services, it’s already expensive enough to run the site. The goal isn’t to stop 100% of scrapers, it was to reduce server load to a level that wasn’t killing the site.
  - jowea 1y ago
    You want them to pay the server costs to serve content to AI scrappers for free? The alternative is Anubis, which is maybe equally annoying to users in a different way.
deleted 1y ago
[deleted]
account42 1y ago
Yay, looking forward to more CAPTCHAs as a regular user.
thephotonsphere 1y ago
account wall :-(
deadbabe 1y ago
No one else can really do this except Cloudflare.
dirkc 1y ago
I assume they will "protect original content online" by blocking LLM clients from ingesting data as context? I'm not optimistic that you can effectively block your original content from ending up in training sets by simply blocking the bots. For now I just assume that anything I put online will end up being used to train some LLM
NullCascade 1y ago
How would you do the opposite of this? Optimize your content to be more likely crawled by AI bots? I know traditional Google-focused SEO is not enough because these AI bots often use other web search/indexing APIs.
- TechDebtDevin 1y ago
  There are script tags you can put in your site from LLM SEO companies if you want your content to be indexed by Perplexity or OpenAI. Theyre kind of too new for me to reccomend.
zargath 1y ago
Sounds very basic, sadly. Anybody know why these web crawling/bot standards are not evolving ? I believe robots.txt was invented in 1994(thx chatgpt). People have tried with sitemaps, RSS and IndexNow, but its like huge$$ organizations are depending on HelloWorld.bas tech to control their entire platform. I want to spin up endpoints/mcp/etc. and let intelligent bots communicate with my services. Let them ask for access, ask for content, pay for content, etc. I want to offer solutions for bots to consume my content, instead of having to choose between full or no access. I am all for AI, but please try to do better. Right now the internet is about to be eaten up by stupid bot farms and served into chat screens. They dont want to refer back to their source and when they do its with insane error rates.
- TechDebtDevin 1y ago
  This comment seems like it comes from a Cloudflare employee. This is clearly the first step in cf building out a marketplace where they will (fail) at attempting to be the middleman in a useless market between crawlers and publishers.
  - zargath 1y ago
    nah, disappointed cf customer
- stereolambda 1y ago
  > I believe robots.txt was invented in 1994(thx chatgpt). Not to pick on you, but I find it quicker to open new tab and do "!w robots.txt" (for search engines supporting the bang notation) or "wiki robots.txt"<click> (for Google I guess). The answer is right there, no need to explain to LLM what I want or verify [1]. [1] Ok, Wikipedia can be wrong, but at least it is a commonly accessible source of wrong I can point people to if they call me out. Plus my predictive model of Wikipedia wrongness gives me pretty low likelihood for something like this, while for ChatGPT it is more random.
- reaperducer 1y ago
  robots.txt was invented in 1994(thx chatgpt) Thought of and discussed as a possibility in 1994. Proposed as a standard in 2019. Adopted as a standard in 2022. Thanks, IETF.
StochasticLi 1y ago
ehem https://github.com/Kaliiiiiiiiii-Vinyzu/patchright https://github.com/Kaliiiiiiiiii-Vinyzu/patchright
j45 1y ago
This is interesting. I'm a fan of Cloudflare, and appreciate all the free tiers they put out there for many. Today I see this article about Cloudflare blocking scrapers. There are useful and legitimate cases where I ask Claude to go research something for me. I'm not sure if Cloudflare discerns legitimate search/research traffic from an AI client vs scraping. Of the sites that are blocked by default will include content by small creators (unless on major platforms with deal?), while the big guys who have something to sell like an Amazon, etc, will likely be able to facilitate and afford a deal to show up more in the results. A few days ago, Cloudflare is also looking to charge AI companies to scrape the content, which is cached copies of other people's content. I'm guessing it will involve paying the owners of the data at some point as well. Being able to exclude it from this purpose (sell/license content, or scrape) would be a useful lever. Putting those two stories together: - Is this a new form of showing up in the AISEO (Search everywhere optimization) to show up in an AI's corpus or ability to search the web, or paying licensing fees instead of advertising fees.. these could be new business models which are interesting, but trying to see where these steps may vector ahead towards, and what to think about today. - With training data being the most valuable thing for AI companies, and this is another avenue for revenue for Cloudflare, this can look like a solution which helps with content licensing as a service. I'd like to see where abstracting this out further ends up going Maybe I'm missing something, is anyone else seeing it this way, or another way that's illuminating to them? Is anyone thinking about rolling their own service for whatever parts of Cloudflare they're using?
- ec109685 1y ago
  It seems like search access is more valuable these days since reasoning requires realtime access to site data.
  - j45 1y ago
    Owning demand for X is what a lot of startups are about. I think Sparktoro had some information that the majority of people are not doing search on AI, and it's still web 90%+ or something like that.
ssijak 1y ago
I dont want this by default. I want my website to end up in AI chatbots. For SEO
abalashov 1y ago
Few people realise that virtually everything we do online has, until this point, been free training to make OpenAI, Anthropic, etc. richer while cutting humans--the ones who produced the value--out of the loop. It might be too little, too late, at this juncture, and this particular solution doesn't seem too innovative. However, it is directionally 100% correct, and let's hope for massively more innovation in defending against AI parasitism.
- k__ 1y ago
  Is anyone suing to make the models and their weights open source?
- jefftk 1y ago
  I write online (comments here, open source software, blogging, etc) because I have ideas I want to share. Whether it's "I did a thing and here's how" or "we should change policy in this specific way" or "does anyone know how to X" I'm happy for this to go into training models just like I'm happy for it to go into humans reading.
  - dolebirchwood 1y ago
    Thank you for having this attitude. I have never attempted any blogging because I always figured no one is actually going to read it. With LLMs, however, I know they will. I actually see this as a motivation to blog, as we are in a position to shape this emerging knowledge base. I don't find it discouraging that others may be profiting off our freely published work, just as I myself have benefited tremendously from open source and the freely published works of others.
    - arkmm 1y ago
      This is an interesting take, thanks for sharing. I wonder how someone should adjust their blogging if they believe their primary audience will be LLMs.
      - 1y ago
jjangkke 1y ago
so TLDR it adjusts your robot.txt and relies on cloudflare to catch bot behavior and it doesn't actually do any sophisticated residential proxy filtering or common bypass methods that works on cloudflare turnstill, do I have this correct? this just pushes AI agents "underground" to adopt the behavior of a full blown stealth focused scraper which makes it harder to detect.
nemild 1y ago
Think this is the future, as the AI Web takes over the human web. At Coinbase, we've been building tools to make the blockchain the ideal payment rails for use cases like this with our x402 protocol: https://www.x402.org/ https://www.x402.org/ Ping if you're interested in joining our open source community.
bgwalter 1y ago
The destruction of the Web and IP theft needs to be addressed legally. The opinion of a single judge notwithstanding, "AI" scraping already violates copyright. This needs to be made explicit in law and scrapers must get the same treatment as Western governments gave to thousands of individuals who were bankrupted or jailed for copyright infringement. We are in the Napster phase of Web content stealing.
zackmorris 1y ago
As usual, this is the wrong approach. The open web is akin to the commons, public domain and public land. So this is like putting a spy cam on a freeway billboard, detecting autonomous vehicles, and shining a spotlight at their camera to block them from seeing the ad. To what end? Eventually these questions will need to be decided in court: 1) Do netizens have the right to anonymity? If not, then we'll have to disclose whether we're humans or artificial beings. Spying on us and blocking us on a whim because our behavior doesn't match social norms will amount to an invasion of privacy (eventually devolving into papers please). 2) Is blocking access to certain users discrimination? If not, then a state-sanctioned market of civil rights abuse will grow around toll roads (think whites-only drinking fountains). 3) Is downloading copyrighted material for learning purposes by AI or humans the same as pirating it and selling it for profit? If so, then we will repeat the everyone-is-a-criminal torrenting era of the 2000s and 2010s when "making available" was treated the same as profiting from piracy, and take abuses by HBO, the RIAA/MPAA and other organizations who shut off users' internet connections through threat of legal actions like suing for violating the DMCA (which should not have been made law in the first place). I'm sure there are more. If we want to live in a free society, then we must be resolute in our opposition of draconian censorship practices by private industry. Gatekeeping by large, monopolistic companies like Cloudflare simply cannot be tolerated. I hope that everyone who reads this finds alternatives to Cloudflare and tells their friends. If they insist on pursuing this attack on our civil rights for profit, then I hope we build a countermovement by organizing with the EFF and our elected officials to eventually bring Cloudflare up on antitrust charges. Cloudflare has shown that they lack the judgement to know better. Which casts doubt on their technical merits and overall vision for how the internet operates. By pursuing this course of action, they have lost face like Google did when it removed its "don't be evil" slogan from its code of conduct so it could implement censorship and operate in China (among other ensh@ttification-related goals). Edit: just wanted to add that I realize this may be an opt-in feature. But that's not the point - what I'm saying is that this starts a bad precedent and an unnecessary arms race, when we should be questioning whether spidering and training AI on copyrighted materials are threats in the first place.
sct202 1y ago
My data served by Cloudflare has increased to 100gb /month compared to <20gb like 2 years ago, and they're all fairly static hobby sites. Actual people traffic is down by like half in the same time frame, so I imagine a lot of this is probably cost savings for Cloudflare to reduce resource usage.
- Apofis 1y ago
  Makes total sense, bandwidth on this scale is expensive.
e38383 1y ago
Why is every second article about this claiming that it’s automatic? It needs to be turned on or at least there was no mention of automatic in the original blog post. I really hope that we can continue training AI the same way we train humans – basically for free.
aunty_helen 1y ago
I saw yesterday that they were going to allow websites to charge per scrape. Looks like cloudflare just invented the new App Store.
hmate9 1y ago
Isn’t this only useful for blogs, news sites, or forums? Why would I want an AI to know less about my product? I want it to understand it, talk about it, and ideally recommend it. Should be default off.
maximilianburke 1y ago
Every evolution of the web, from Web 2 giving us walled gardens to Web 3 giving us, well, nothing, to what we have now is taking us further from a network of communities and personal repositories of knowledge. Sure, fidelity has gotten better but so much has been lost.
sneak 1y ago
This idea that you can publish data for people to download and read but not for people to download and store, or print, or think about, or train on is a doomed one. If you don’t want people reading your data, don’t put it on the web. The concept that copyright extends to “human eyeballs only” is a silly one.
- t1001 1y ago
  With the problem being bots hammering the site en masse, it feels like the better analog is "allowing free replicator use without having someone ruin the fun by requesting ten tons of food be produced in their quarters every minute".
ChrisArchitect 1y ago
[dupe] https://news.ycombinator.com/item?id=44432385 https://news.ycombinator.com/item?id=44432385
kristoff200512 1y ago
AI will endlessly crawl my website, quickly exhausting the egress quota of my Supabase free plan,but Cloudflare can stop all of this.
userbinator 1y ago
s/A.I. Data Scrapers/non-sanctioned browsers running on non-sanctioned platforms/ They've been trying to do this for years. Now "AI" gives a convenient excuse.
nsoonhui 1y ago
But how is this effective against Gemini and even OpenAI, who can instead of relying on their Google and Bing crawlers respectively to crawl the content?
deleted 1y ago
[deleted]

429 points by stephendause 1y ago

YPPH 1y ago

This is great. But my concerns about Cloudflare's power remain. Today it's blocking AI crawlers, tomorrow will it be blocking all browsers that fail hardware-attestation checks?

cmg 1y ago

Archive link: https://archive.ph/ARnyu https://archive.ph/ARnyu

joshdavham 1y ago
How did you make that link?

badlibrarian 1y ago

Did they ever fix the auto-blocking of RSS feeds? https://news.ycombinator.com/item?id=41864632 https://news.ycombinator.com/item?id=41864632

blakesterz 1y ago

The list of bots is pretty short right now: https://developers.cloudflare.com/bots/concepts/bot/#ai-bots https://developers.cloudflare.com/bots/concepts/bot/#ai-bots

ZiiS 1y ago
Enough to more than half the traffic to most sites if the blocks hold.
hennell 1y ago
Cloudflare sees a lot of the web traffic. I assume these are the biggest bots they're seeing right now, and any new contenders would be added as they find them. Probably impossible to really block everything, but they've got the web-coverage to detect more than most.
- TechDebtDevin 1y ago
  They are lying. They cant detect crawlers unless we tell them we are who we are.
JimDabell 1y ago
> AI bots > You can opt into a managed rule that will block bots that we categorize as artificial intelligence (AI) crawlers (“AI Bots”) from visiting your website. Customers may choose to do this to prevent AI-related usage of their content, such as training large language models (LLM). > CCBot (Common Crawl) Common Crawl is not an AI bot: https://commoncrawl.org https://commoncrawl.org
- johneth 1y ago
  The data it collects is used by AI companies, though.

Spivak 1y ago

Poor ChatGPT-User, nobody understands you. Blocking a real user because of the, admittedly odd, browser they're using misses the point.

Roark66 1y ago

This is a bit silly. Slowing down, yes, but blocking? People who *really* want that content will find a way and this will hit everyone else instead that will have to do silly riddles before following every link or run crypto mining for them before being shown the content . I recently went to a big local auction site on which I buy frequently and I got one of these "we detected unusual traffic from your network" messages. And "prove you're human". Which was followed by "you completed the capcha in 0.4s your IP is banned". Really? Am I supposed to slow down my browsing now? I tried a different browser, a different OS, logging on,clearing cookies, etc. Same result when I tried a search. It took 4h after contacting their customer service to unblock it. And the explanation was "you're clicking too fast". At some point it just becomes a farce and the hassle is not worth the content. Also, while my story doesn't involve any bots perhaps a time will come when local LLMs will be good enough that I'll be able to tell one "reorder my cat food" and it will go and do it. Why are they so determined to "stop it" (spoiler, they can't). For anyone who says LLMs are already capable of ordering cat food I say not so fast. First the cat food has to be on sale/offer (sometimes combined with extras). Second it is supposed to be healthy (no grains) and third the taste needs to be to my cats liking. So far I'm not going to trust a LLM with this.

picohernandez 1y ago
I was chatting with my sister last weekend. As a hobby, she creates and sells wedding invitation and other designs at an online marketplace site called Zazzle. She was telling me all about how that site implemented some automatic bot detection and it sounded like it was a total disaster. Real content creators were getting wrongly flagged as bots and then getting blocked from using the site just for using the most fundamental site functionality, and to make it worse, then it was apparently impossible for them to get past the captcha challenge or whatever it showed next. She forwarded me a link to some support forum discussion about it and it was mindboggling the troubles that some of the content creators there had to go through: https://community.zazzle.com/t5/technical-issues/bot-test-won-t-let-me-into-my-account-press-and-hold-didn-t-work/m-p/205396 https://community.zazzle.com/t5/technical-issues/bot-test-wo... My sister said that her sales figures are way down compared to what they used to be and she didn't know if this bot flagger was disrupting real paying customers too. She said it had flagged her a couple of times, although she was luckily able to get past the bot challenge. She has pretty much given up on making and uploading new designs because of what was happening to other content creators there. She's now scared to use the site because she doesn't want to get wrongly locked out of her account.

Sol- 1y ago

Do the major AI companies actually honor robots.txt? Even if some of their publicly known crawlers might do it, surely they have surreptitious campaigns where they do some hidden crawling, just like how they illegally pirate books, images and user data to train on.

px43 1y ago
There's a lack of clarity, but it seems likely to me that a majority of this traffic is actually people asking questions to the AI, and the AI going out and researching for answers. When the AI tools are being used like a web browser to do research, should they still be adhering to robots.txt, or is that only intended for search indexing?
chasd00 1y ago
My thought too, honoring robots.txt is just a convention. There's no requirement to follow robots.txt, or at least certainly no technical requirement. I don't think there's any automatic legal requirement either. Maybe sites could add "you must honor policies set in robots.txt" to something like a terms of service but I have no idea if that would have enough teeth for a crawler to give up.
- TechDebtDevin 1y ago
  Cloudflare snd their customera have been desperately for years trying to kill scrapers in court. This is all. Meaningless, but they are probably gearing up for another legal battle to define robots.txt as a legal contract. Theyre going to use this marketplace theyre scamming people with to do it. They will fail.
- prmoustache 1y ago
  I don't think terms of service are applicable anyway. Terms of Service aren't a signed contract as you may never see it nor know there is one. This happens both in the case of visiting the site interactively or fetching a page programatically.
mschuster91 1y ago
Cloudflare, for all I hate their role as a gatekeeper these days, actually has the leverage to force the AI companies to bend.

btown 1y ago

The headline is somewhat misleading: sites using Cloudflare now have an opt-in option to quickly block all AI bots, but it won't be turned on by default for sites using Cloudflare. The idea that Cloudflare could do the latter at the sole discretion of its leadership, though, is indicative of the level of power Cloudflare holds.

bitpush 1y ago
It is now an adversarial relationship between aibots and website, and cloudflare is merely reacting to it. Would you say the same for ddos protection? Isn't that the same as well?
- TechDebtDevin 1y ago
  They arent doing anything. They are attempting to insert themselves into the middle of a marketplace (that doesnt exist and never will) where scrapers pay for IP. They think theyre going to profit off the bots, not protect your site. Dont fall for their scam.
  - bitpush 1y ago
    What do you mean they are trying to insert themselves? If I have a website that I host with cloudflare, I (as the rightful website owner) has inserted Cloudflare in between. It isnt CF going around saying, that's a nice website you have there. I'm gonna put myself in between.
    - deleted 1y ago
      [deleted]
GrayShade 1y ago
> sites using Cloudflare now have an opt-in option to quickly block all AI bots, but it won't be turned on by default for sites using Cloudflare Do you have a source for that? https://blog.cloudflare.com/content-independence-day-no-ai-crawl-without-compensation/ https://blog.cloudflare.com/content-independence-day-no-ai-c... does say "changing the default".

postalcoder 1y ago

> When you enable this feature via a pre-configured managed rule, Cloudflare can detect and block verified AI bots that comply with robots.txt and respect crawl rates, and do not hide their behavior from your website. The rule has also been expanded to include more signatures of AI bots that do not follow the rules. We already know companies like Perplexity are masking their traffic. I'm sure there's more than meets the eye, but taking this at face value, doesn't punishing respectful and transparent bots only incentivize obfuscation? edit: This link[0], posted in a comment elsewhere, addresses this question. tldr, obfuscation doesn't work. > We leverage Cloudflare global signals to calculate our Bot Score, which for AI bots like the one above, reflects that we correctly identify and score them as a “likely bot.” > When bad actors attempt to crawl websites at scale, they generally use tools and frameworks that we are able to fingerprint. For every fingerprint we see, we use Cloudflare’s network, which sees over 57 million requests per second on average, to understand how much we should trust this fingerprint. To power our models, we compute global aggregates across many signals. Based on these signals, our models were able to appropriately flag traffic from evasive AI bots, like the example mentioned above, as bots. [0] https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo...

colechristensen 1y ago
>doesn't punishing respectful and transparent bots only incentivize obfuscation? They're cloudflare and it's not like it's particularly easy to hide a bot that is scraping large chunks of the Internet from them. On top of the fact that they can fingerprint any of your sneaky usage, large companies have to work with them so I can only assume there are channels of communication where cloudflare can have a little talk with you about your bad behavior. I don't know how often lawyers are involved but I would expect them to be.
jerf 1y ago
"doesn't punishing respectful and transparent bots only incentivize obfuscation?" Sure, but we crossed that bridge over 20 years ago. It's not creating an arms race where there wasn't already one. Which is my generic response to everyone bringing similar ideas up. "But the bots could just...", yeah, they've been doing it for 20+ years and people have been fighting it for just as long. Not a new problem, not a new set of solutions, no prospect of the arms race ending any time soon, none of this is new.

dougb5 1y ago

> Cloudflare can detect and block verified AI bots that comply with robots.txt and respect crawl rates, and do not hide their behavior from your website It's the bots that do hide their behavior -- via residential proxy services -- that are causing most of the burden, for my site anyway. Not these large commercial AI vendors.

alganet 1y ago

> If A.I. companies freely use data from various websites without permission or payment, people will be discouraged from creating new digital content I don't see a way out of this happening. AI fundamentally discourages other forms of digital interaction as it grows. Its mechanism of growing is killing other kinds of digital content. It will eventually kill the web, which is, ironically, its main source of food.

preachermon 1y ago
[flagged]
- alganet 1y ago
  These kinds of comparisons rarely lead to good discussions. Let's instead be focused and talk about real stuff. Consider https://learnpythonthehardway.org/ https://learnpythonthehardway.org/ for example. It has influenced a generation of Python developers. Not just the main website, but the tons of Python code and Python-related content it inspired. Why would anyone write these kinds of textbooks/websites/guides if AI can replace them? AI companies are effectively broadcasting you don't need the hard way anymore, you can just vibe. Arguibly though, without the existance of Learn Python the Hard Way and similar content, AI would be worse at writing Python stuff. That's what I mean by "main source of food", good content that influences a lot of people. Net-positive effects hard to predict or even identify except for the more popular cases (such as LPTHW). If my prediction is right, no one will notice that good content has stopped being produced. It will appear as if content is being created in generally the same way as before, but in reality, these long tail initiatives like LPTHW will have ceased before anyone can do anything about it. Again, I don't see a way out of this scenario. Not for AI companies, not for content writers. This is going to happen. The world in which I'm wrong is the best one.
  - mfost 1y ago
    In a similar vein, I remember people advocating for replacing new untrained hires with AI. After all, a competent senior engineer is needed to validate the contributions of the new hires anyway and they can do the same checking the AI code. But then, how would you even train and replace those competent seniior engineers that do the filtering when they retire? The whole system was predicated on having a chain of new hires that gain experience in the process.

jasonthorsness 1y ago

I turned this on and it adjusts the robots.txt automatically; not sure what else it is doing. # NOTICE: The collection of content and other data on this # site through automated means, including any device, tool, # or process designed to data mine or scrape content, is # prohibited except (1) for the purpose of search engine indexing or # artificial intelligence retrieval augmented generation or (2) with express # written permission from this site’s operator. # To request permission to license our intellectual # property and/or other materials, please contact this # site’s operator directly. # BEGIN Cloudflare Managed content User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: meta-externalagent Disallow: / # END Cloudflare Managed Content User-agent: * Disallow: /* Allow: /$

xyst 1y ago
So in addition to updating the robots.txt file, which really only blocks a small number of them. Seems CF has been gathering data and profiling these malicious agents. This post by CF elaborates a bit further: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo... Basically becomes a game of cat and mouse.
postalcoder 1y ago
This is interesting. The reasoning and response don't line up. > Cloudflare is making the change to protect original content on the internet, Mr. Prince said. If A.I. companies freely use data from various websites without permission or payment, people will be discouraged from creating new digital content, he said > prohibited except for the purpose of [..] artificial intelligence retrieval augmented generation This seems to be targeted at taxing training of language models, but why an exclusion for the RAG stuff? That seems like it has a much greater immediate impact for online content creators, for whom the bots are obviating a click.
- fennecfoxy 1y ago

cratermoon 1y ago

I'm still not sure this is going to be very effective, as so many of the worst offenders don't identify themselves as bots, and often change their user agent. Has Cloudflare said anything about identifying the bad actors?

chasd00 1y ago
i've mentioned this in a couple replies so maybe i'm wrong but it's up to the client to obey robots.txt. Why would they not just ignore it? Unless there's some legal consequence not complying with robots.txt then why even follow it? There's no technical enforcement of the policies in the file, it's up to the client to honor them.
- kentonv 1y ago
  > There's no technical enforcement of the policies in the file, it's up to the client to honor them. That's incorrect. Cloudflare does in fact enforce this at a technical level. Cloudflare has been doing bot detection for years and can pretty reliably detect when bots are not following robots.txt and then block them.
GrayShade 1y ago
Yes, they have over the years, for example https://blog.cloudflare.com/residential-proxy-bot-detection-using-machine-learning/ https://blog.cloudflare.com/residential-proxy-bot-detection-..., https://blog.cloudflare.com/cloudflare-bot-management-machine-learning-and-more/ https://blog.cloudflare.com/cloudflare-bot-management-machin..., https://blog.cloudflare.com/introducing-bot-analytics/ https://blog.cloudflare.com/introducing-bot-analytics/.
- cratermoon 1y ago
  Between those measures, if they are effective and the new blocking, maybe the bigger companies will be induced to behave a little better.

lucasyvas 1y ago

I fail to see how this won’t just result in UA string or other obfuscation.

chasd00 1y ago
a crawler doesn't have to change anything, they can just ignore the robots.txt file. It's up to the client to read robots.txt and follow its directives but there's no technical reason why the client cannot just ignore everything in the file period.
kube-system 1y ago
Cloudflare’s filtering is already way more sophisticated than just looking at UA string or other voluntary reporting. They’re almost certainly using fingerprinting and behavioral analytics.

gazpacho 1y ago

From an open source projects perspective we’d want to disable this on our docs sites. We actually want those to be very discoverable by LLMs, during training or online usage.

rorylaitila 1y ago

Unfortunately I think pissing into the wind. Information websites are all but dead. AI contains all published human information. If you have positioned your website as an answer to a question, it won't survive that way. "Information" is dead but content is not. Stories, empathy, community, connection, products, services. Content of this variety is exploding. The big challenge is discoverability. Before, information arbitrage was one pathway to get your content discovered, or to skim a profit. This is over with AI. New means of discovery are necessary, largely network and community based. AI will throw you a few bones, but it will be 10% of what SEO did.

fennecfoxy 1y ago
>AI contains all published human information No, it most certainly does not. It was certainly trained on large swathes of human knowledge/interactions. A model that consists of a perfect representation/compression of all this info is a zip file, not a model file.
- rorylaitila 1y ago
  AI providers have scrapped and will continue to, all internet published information or virtually so. Since "Information" is infinite, AI cannot contain "all information" in a complete sense. But it certainly answers almost everything that matters for any existing search query that has ever been targeted by a webpage that is crawlable. In any case, as manifest by real world SEO, which is plummeting in traffic for informational queries, the effect is the same. This real world impact is what matters and will not be reversed, regardless of attempts at blocking.
ozgrakkurt 1y ago
You are assuming LLMs will replace search engines. Why is this the case? To me it seems like there has to be so much optimization for this to happen that, it is not likely. LLM answers are slow and unreliable. Even using something like perplexity doesn’t give much value over using a regular search engine in my experience
- 1y ago

Meekro 1y ago

I've heard lots of people on HN complaining about bot traffic bogging down their websites, and as a website operator myself I'm honestly puzzled. If you're already using Cloudflare, some basic cache configuration should guarantee that most bot traffic hits the cache and doesn't bog down your servers. And even if you don't want to do that, bandwidth and CPU are so cheap these days that it shouldn't make a difference. Why is everyone so upset?

deepsiml 1y ago
Not much into that kind of DevOps. What is a good basic caching in this instance?
- TechDebtDevin 1y ago
  Cloudflare and other CDNs will usually automatically cache your static pages.
- haiku2077 1y ago
  It comes down to: 1. Use the Cache-Control header to express how to cache your site correctly (https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Caching https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Cac...) 2. Use a CDN service, or at least a caching reverse proxy, to serve most of the cacheable requests to reduce load on the (typically much more expensive) origin servers
  - mrweasel 1y ago
    Just note that many AI scrapers will go to great length to do cache busting. For some reason many of them feel like they need to get the absolute latest version and don't trust your cache.
    - haiku2077 1y ago
      You can use Cache Control headers to express that your own CDN should aggressively refresh a resource but always serve it to external clients from cache. It's covered in the link under "Managed Caches"

deleted 1y ago

[deleted]

yodon 1y ago

Discussed yesterday (270+ comments)[0] [0]https://news.ycombinator.com/item?id=44432385 https://news.ycombinator.com/item?id=44432385

dawnerd 1y ago

I’ve been using this for a while on my mastodon server and after a few tweaks to make sure it wasn’t blocking legit traffic it’s been really working great. Between Microsoft and Meta, they were hitting my services more than any other traffic combined which says a lot of you know how noisy mastodon can be. Server load went down dramatically. It also completely put a stop to perplexity as far as I can tell. And the robots file meant nothing, they’d still request it hundreds of thousands of times instead of caching it. Every request they’d hit it first then hit their intended url.

TechDebtDevin 1y ago
This does nothing dude. Literally nothing. OpenAI or whoever are just going to hire people like me who dont get caught. Stop ruining the experience of users and allowing cf to fill the internet with more bloated javascript challenge pages and privacy invading fingerprinting. Stop making cf the police of the internet. We're literally handing the internet to this company on a silver platter to do MITM attacks on our privacy and god knows what else. Fucking wild.
- fluidcruft 1y ago
  They literally said it significantly reduced their server resource usage. Are you suggesting they are lying?
- drowsspa 1y ago
  Why do you think you have the moral high ground here?
- dawnerd 1y ago
  Well the alternative is to not have an instance at all so… what do you suggest? I’m not paying for the other services, it’s already expensive enough to run the site. The goal isn’t to stop 100% of scrapers, it was to reduce server load to a level that wasn’t killing the site.
- jowea 1y ago
  You want them to pay the server costs to serve content to AI scrappers for free? The alternative is Anubis, which is maybe equally annoying to users in a different way.

deleted 1y ago

[deleted]

account42 1y ago

Yay, looking forward to more CAPTCHAs as a regular user.

thephotonsphere 1y ago

account wall :-(

deadbabe 1y ago

No one else can really do this except Cloudflare.

dirkc 1y ago

I assume they will "protect original content online" by blocking LLM clients from ingesting data as context? I'm not optimistic that you can effectively block your original content from ending up in training sets by simply blocking the bots. For now I just assume that anything I put online will end up being used to train some LLM

NullCascade 1y ago

How would you do the opposite of this? Optimize your content to be more likely crawled by AI bots? I know traditional Google-focused SEO is not enough because these AI bots often use other web search/indexing APIs.

TechDebtDevin 1y ago
There are script tags you can put in your site from LLM SEO companies if you want your content to be indexed by Perplexity or OpenAI. Theyre kind of too new for me to reccomend.

zargath 1y ago

Sounds very basic, sadly. Anybody know why these web crawling/bot standards are not evolving ? I believe robots.txt was invented in 1994(thx chatgpt). People have tried with sitemaps, RSS and IndexNow, but its like huge$$ organizations are depending on HelloWorld.bas tech to control their entire platform. I want to spin up endpoints/mcp/etc. and let intelligent bots communicate with my services. Let them ask for access, ask for content, pay for content, etc. I want to offer solutions for bots to consume my content, instead of having to choose between full or no access. I am all for AI, but please try to do better. Right now the internet is about to be eaten up by stupid bot farms and served into chat screens. They dont want to refer back to their source and when they do its with insane error rates.

TechDebtDevin 1y ago
This comment seems like it comes from a Cloudflare employee. This is clearly the first step in cf building out a marketplace where they will (fail) at attempting to be the middleman in a useless market between crawlers and publishers.
- zargath 1y ago
  nah, disappointed cf customer
stereolambda 1y ago
> I believe robots.txt was invented in 1994(thx chatgpt). Not to pick on you, but I find it quicker to open new tab and do "!w robots.txt" (for search engines supporting the bang notation) or "wiki robots.txt"<click> (for Google I guess). The answer is right there, no need to explain to LLM what I want or verify [1]. [1] Ok, Wikipedia can be wrong, but at least it is a commonly accessible source of wrong I can point people to if they call me out. Plus my predictive model of Wikipedia wrongness gives me pretty low likelihood for something like this, while for ChatGPT it is more random.
reaperducer 1y ago
robots.txt was invented in 1994(thx chatgpt) Thought of and discussed as a possibility in 1994. Proposed as a standard in 2019. Adopted as a standard in 2022. Thanks, IETF.

StochasticLi 1y ago

ehem https://github.com/Kaliiiiiiiiii-Vinyzu/patchright https://github.com/Kaliiiiiiiiii-Vinyzu/patchright

j45 1y ago

This is interesting. I'm a fan of Cloudflare, and appreciate all the free tiers they put out there for many. Today I see this article about Cloudflare blocking scrapers. There are useful and legitimate cases where I ask Claude to go research something for me. I'm not sure if Cloudflare discerns legitimate search/research traffic from an AI client vs scraping. Of the sites that are blocked by default will include content by small creators (unless on major platforms with deal?), while the big guys who have something to sell like an Amazon, etc, will likely be able to facilitate and afford a deal to show up more in the results. A few days ago, Cloudflare is also looking to charge AI companies to scrape the content, which is cached copies of other people's content. I'm guessing it will involve paying the owners of the data at some point as well. Being able to exclude it from this purpose (sell/license content, or scrape) would be a useful lever. Putting those two stories together: - Is this a new form of showing up in the AISEO (Search everywhere optimization) to show up in an AI's corpus or ability to search the web, or paying licensing fees instead of advertising fees.. these could be new business models which are interesting, but trying to see where these steps may vector ahead towards, and what to think about today. - With training data being the most valuable thing for AI companies, and this is another avenue for revenue for Cloudflare, this can look like a solution which helps with content licensing as a service. I'd like to see where abstracting this out further ends up going Maybe I'm missing something, is anyone else seeing it this way, or another way that's illuminating to them? Is anyone thinking about rolling their own service for whatever parts of Cloudflare they're using?

ec109685 1y ago
It seems like search access is more valuable these days since reasoning requires realtime access to site data.
- j45 1y ago
  Owning demand for X is what a lot of startups are about. I think Sparktoro had some information that the majority of people are not doing search on AI, and it's still web 90%+ or something like that.

ssijak 1y ago

I dont want this by default. I want my website to end up in AI chatbots. For SEO

abalashov 1y ago

Few people realise that virtually everything we do online has, until this point, been free training to make OpenAI, Anthropic, etc. richer while cutting humans--the ones who produced the value--out of the loop. It might be too little, too late, at this juncture, and this particular solution doesn't seem too innovative. However, it is directionally 100% correct, and let's hope for massively more innovation in defending against AI parasitism.

k__ 1y ago
Is anyone suing to make the models and their weights open source?
jefftk 1y ago
I write online (comments here, open source software, blogging, etc) because I have ideas I want to share. Whether it's "I did a thing and here's how" or "we should change policy in this specific way" or "does anyone know how to X" I'm happy for this to go into training models just like I'm happy for it to go into humans reading.
- dolebirchwood 1y ago
  Thank you for having this attitude. I have never attempted any blogging because I always figured no one is actually going to read it. With LLMs, however, I know they will. I actually see this as a motivation to blog, as we are in a position to shape this emerging knowledge base. I don't find it discouraging that others may be profiting off our freely published work, just as I myself have benefited tremendously from open source and the freely published works of others.
  - arkmm 1y ago
    This is an interesting take, thanks for sharing. I wonder how someone should adjust their blogging if they believe their primary audience will be LLMs.
    - 1y ago

jjangkke 1y ago

so TLDR it adjusts your robot.txt and relies on cloudflare to catch bot behavior and it doesn't actually do any sophisticated residential proxy filtering or common bypass methods that works on cloudflare turnstill, do I have this correct? this just pushes AI agents "underground" to adopt the behavior of a full blown stealth focused scraper which makes it harder to detect.

nemild 1y ago

Think this is the future, as the AI Web takes over the human web. At Coinbase, we've been building tools to make the blockchain the ideal payment rails for use cases like this with our x402 protocol: https://www.x402.org/ https://www.x402.org/ Ping if you're interested in joining our open source community.

bgwalter 1y ago

The destruction of the Web and IP theft needs to be addressed legally. The opinion of a single judge notwithstanding, "AI" scraping already violates copyright. This needs to be made explicit in law and scrapers must get the same treatment as Western governments gave to thousands of individuals who were bankrupted or jailed for copyright infringement. We are in the Napster phase of Web content stealing.

zackmorris 1y ago

As usual, this is the wrong approach. The open web is akin to the commons, public domain and public land. So this is like putting a spy cam on a freeway billboard, detecting autonomous vehicles, and shining a spotlight at their camera to block them from seeing the ad. To what end? Eventually these questions will need to be decided in court: 1) Do netizens have the right to anonymity? If not, then we'll have to disclose whether we're humans or artificial beings. Spying on us and blocking us on a whim because our behavior doesn't match social norms will amount to an invasion of privacy (eventually devolving into papers please). 2) Is blocking access to certain users discrimination? If not, then a state-sanctioned market of civil rights abuse will grow around toll roads (think whites-only drinking fountains). 3) Is downloading copyrighted material for learning purposes by AI or humans the same as pirating it and selling it for profit? If so, then we will repeat the everyone-is-a-criminal torrenting era of the 2000s and 2010s when "making available" was treated the same as profiting from piracy, and take abuses by HBO, the RIAA/MPAA and other organizations who shut off users' internet connections through threat of legal actions like suing for violating the DMCA (which should not have been made law in the first place). I'm sure there are more. If we want to live in a free society, then we must be resolute in our opposition of draconian censorship practices by private industry. Gatekeeping by large, monopolistic companies like Cloudflare simply cannot be tolerated. I hope that everyone who reads this finds alternatives to Cloudflare and tells their friends. If they insist on pursuing this attack on our civil rights for profit, then I hope we build a countermovement by organizing with the EFF and our elected officials to eventually bring Cloudflare up on antitrust charges. Cloudflare has shown that they lack the judgement to know better. Which casts doubt on their technical merits and overall vision for how the internet operates. By pursuing this course of action, they have lost face like Google did when it removed its "don't be evil" slogan from its code of conduct so it could implement censorship and operate in China (among other ensh@ttification-related goals). Edit: just wanted to add that I realize this may be an opt-in feature. But that's not the point - what I'm saying is that this starts a bad precedent and an unnecessary arms race, when we should be questioning whether spidering and training AI on copyrighted materials are threats in the first place.

sct202 1y ago

My data served by Cloudflare has increased to 100gb /month compared to <20gb like 2 years ago, and they're all fairly static hobby sites. Actual people traffic is down by like half in the same time frame, so I imagine a lot of this is probably cost savings for Cloudflare to reduce resource usage.

Apofis 1y ago
Makes total sense, bandwidth on this scale is expensive.

e38383 1y ago

Why is every second article about this claiming that it’s automatic? It needs to be turned on or at least there was no mention of automatic in the original blog post. I really hope that we can continue training AI the same way we train humans – basically for free.

aunty_helen 1y ago

I saw yesterday that they were going to allow websites to charge per scrape. Looks like cloudflare just invented the new App Store.

hmate9 1y ago

Isn’t this only useful for blogs, news sites, or forums? Why would I want an AI to know less about my product? I want it to understand it, talk about it, and ideally recommend it. Should be default off.

maximilianburke 1y ago

Every evolution of the web, from Web 2 giving us walled gardens to Web 3 giving us, well, nothing, to what we have now is taking us further from a network of communities and personal repositories of knowledge. Sure, fidelity has gotten better but so much has been lost.

sneak 1y ago

This idea that you can publish data for people to download and read but not for people to download and store, or print, or think about, or train on is a doomed one. If you don’t want people reading your data, don’t put it on the web. The concept that copyright extends to “human eyeballs only” is a silly one.

t1001 1y ago
With the problem being bots hammering the site en masse, it feels like the better analog is "allowing free replicator use without having someone ruin the fun by requesting ten tons of food be produced in their quarters every minute".

ChrisArchitect 1y ago

[dupe] https://news.ycombinator.com/item?id=44432385 https://news.ycombinator.com/item?id=44432385

kristoff200512 1y ago

AI will endlessly crawl my website, quickly exhausting the egress quota of my Supabase free plan,but Cloudflare can stop all of this.

userbinator 1y ago

s/A.I. Data Scrapers/non-sanctioned browsers running on non-sanctioned platforms/ They've been trying to do this for years. Now "AI" gives a convenient excuse.

nsoonhui 1y ago

But how is this effective against Gemini and even OpenAI, who can instead of relying on their Google and Bing crawlers respectively to crawl the content?

deleted 1y ago

[deleted]