8 ms·

> After 15 minutes of confusion, it turned out Cloudflare had put a crazy robots.txt on my site without my consent (Cloudflare, love you guys, but this needs to

> After 15 minutes of confusion, it turned out Cloudflare had put a crazy robots.txt on my site without my consent (Cloudflare, love you guys, but this needs to stop). Might be the first time I see someone complain about their website being protected from a scraper, instead of the other way around.

nibbleyou 2mo ago
I think the issue is the lack of consent. Whether a service I use is protecting my website from scrapers or feeding everything to scrapers, some of us would prefer that it takes our informed consent before doing so.
- bfjvibybd6cuvu6 2mo ago
  It does.
- dannyw 2mo ago
  Cloudflare is explicitly a service for dropping requests, whether it’s DDoS attacks, as a WAF, or AI crawlers. It offers a lot more too, but this isn’t Cloudflare overstepping imo. FWIW, I just set up a domain last week, and the web UI asked if I want to block AI crawlers or not. Perhaps OP set it up agentically, and the agent didn’t pass an optional param correctly, or ticked the box for him?
sva_ 2mo ago
I am pretty sure you have to enable cloudflare to manage your robots.txt, it shouldn't be doing that by itself. Maybe they did it by accident, it is just 1 click.
- zx76 2mo ago
  My understanding is that the current default allows AI training bots - but this actually going to change in 2 months time. https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/ https://developers.cloudflare.com/changelog/post/2026-07-01-... From Sept 15 all new sites added to CF will even block Googlebot by default on any page that serves ads as I understand it. I think it's CF trying to force Google to separate out their bot traffic into bots for training and bots for the search index. I think CF sees a big opportunity to get businesses to pay them to allow certain uses of their data but block others. They're also starting a registry of "Approved" crawlers.
el_io 2mo ago
You have to enable this, or atleast was the case when I tried less than 1 month ago.
- dannyw 2mo ago
  You still have to enable this as of one week ago. I suspect OP might have agentically set up a new Cloudflare domain, and who knows what the agent did during onboarding.
rgrieselhuber 2mo ago
It should still be done with consent. Robots.txt files don’t protect against determined scrapers anyway.
- adrian17 2mo ago
  I thought bot protection was one of reasons for using Cloudflare in the first place (next to general CDN hosting)? After all, they do show a captcha-like challenge on some websites, so I thought that even without robots.txt, it still would have prevented the automated request by default.

nibbleyou 2mo ago

I think the issue is the lack of consent. Whether a service I use is protecting my website from scrapers or feeding everything to scrapers, some of us would prefer that it takes our informed consent before doing so.

bfjvibybd6cuvu6 2mo ago
It does.
dannyw 2mo ago
Cloudflare is explicitly a service for dropping requests, whether it’s DDoS attacks, as a WAF, or AI crawlers. It offers a lot more too, but this isn’t Cloudflare overstepping imo. FWIW, I just set up a domain last week, and the web UI asked if I want to block AI crawlers or not. Perhaps OP set it up agentically, and the agent didn’t pass an optional param correctly, or ticked the box for him?

sva_ 2mo ago

I am pretty sure you have to enable cloudflare to manage your robots.txt, it shouldn't be doing that by itself. Maybe they did it by accident, it is just 1 click.

zx76 2mo ago
My understanding is that the current default allows AI training bots - but this actually going to change in 2 months time. https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/ https://developers.cloudflare.com/changelog/post/2026-07-01-... From Sept 15 all new sites added to CF will even block Googlebot by default on any page that serves ads as I understand it. I think it's CF trying to force Google to separate out their bot traffic into bots for training and bots for the search index. I think CF sees a big opportunity to get businesses to pay them to allow certain uses of their data but block others. They're also starting a registry of "Approved" crawlers.

el_io 2mo ago

You have to enable this, or atleast was the case when I tried less than 1 month ago.

dannyw 2mo ago
You still have to enable this as of one week ago. I suspect OP might have agentically set up a new Cloudflare domain, and who knows what the agent did during onboarding.

rgrieselhuber 2mo ago

It should still be done with consent. Robots.txt files don’t protect against determined scrapers anyway.

adrian17 2mo ago
I thought bot protection was one of reasons for using Cloudflare in the first place (next to general CDN hosting)? After all, they do show a captcha-like challenge on some websites, so I thought that even without robots.txt, it still would have prevented the automated request by default.