> My issue is that crawlers aren’t respecting robots.txt Cloudflare has a toggle switch to automatically block LLM's + scrapers etc: https://blog.cloudflare.c

> My issue is that crawlers aren’t respecting robots.txt Cloudflare has a toggle switch to automatically block LLM's + scrapers etc: https://blog.cloudflare.com/declaring-your-aindependence-block-ai-bots-scrapers-and-crawlers-with-a-single-click/ https://blog.cloudflare.com/declaring-your-aindependence-blo...