6 ms·
I can definitively say llms.txt is not used by any AI players. I run a blogging platform with around 80k blogs and /llms.txt is not requested by anything (other
by HermanMartinus 3mo ago
I can definitively say llms.txt is not used by any AI players. I run a blogging platform with around 80k blogs and /llms.txt is not requested by anything (other than humans checking to see if there's an llms.txt path).
All regular pages are aggressively scraped to the extent it's a problem I have to consistently manage, but not llms.txt.
- isaachinman 3mo agoHow is a static blog being scraped a problem? Do you not use a CDN?
- the_real_cher 3mo agoAre all blogs static though?
- johannes1234321 3mo agoVery few blogs require frequent updates. Even with user comments.
- nickserv 3mo ago> a blogging platform with around 80k blogs But nah, I'm sure OP doesn't know about CDNs.
- nickserv 3mo agoI'm seeing quite a bit of request for these on my work's GitBook documentation site. But perhaps these are developers specifically targeting these pages to feed whatever LLM they are using.
- 0123456789ABCDE 3mo ago> I can definitively say llms.txt is not used by any AI players. https://developers.openai.com/llms.txt https://docs.anthropic.com/llms.txt https://geminicli.com/llms.txt https://github.com/llms.txt https://docs.aws.amazon.com/llms.txt https://openrouter.ai/docs/llms.txt
- m4tthumphrey 3mo agoOP clearly meant that the AI players are not reading and/or honouring llms.txt of other websites when scraping.
- 0123456789ABCDE 3mo agoi stand corrected, but what was clear to you, obviously was not clear to me.
- sunshine-o 3mo agoAmazing, I didn't know. So it get even stranger, I am the only one reading those /llms.txt ...