7 ms·
Meta is an exception. They intentionally ignore robots.txt rules. (No second chances, I blocked their whole IP range.)
by Symbiote 17d ago
Meta is an exception. They intentionally ignore robots.txt rules.
(No second chances, I blocked their whole IP range.)
- inigyou 17d agoEveryone ignores robots.txt because every robots.txt blocks everyone.
- nubinetwork 17d agoI allow Google, Bing, Yahoo, Kagi, and Apple on my personal blog... who else do I need to allow? Claude, OpenAI, Alibaba, and perplexity can go to hell.
- inigyou 17d agoThink about incentives, what benefit does Anthropic obtain by going to hell?