9 ms·
LLMs.txt is also nonsense since it isn't adopted by any of the major AI players.
by rickette 3mo ago
LLMs.txt is also nonsense since it isn't adopted by any of the major AI players.
- dspillett 3mo agoThe same could be said of robots.txt And anything else that might tell them not to access something.
- reddalo 3mo agorobots.txt predates the modern web though
- dspillett 3mo agoMy point was that llms.txt not working is no different from them ignoring everything else that came before and probably everything that is yet to come. If they want it, they will take it, polite directives in text files will have no effect.
- networked 3mo agoGoogle has recently added `llms.txt` to Chrome's Lighthouse check for agentic browsing (https://searchengineland.com/google-llms-txt-chrome-lighthouse-478246 https://searchengineland.com/google-llms-txt-chrome-lighthou...), so adoption may be coming. Admittedly, I put more faith in <link rel="alternate" type="text/markdown" href="https://example.com/foo.md" title="Markdown version of the <Foo> page"> that I copied from Gwern.net. This convention is discoverable (just read the HTML) and naturally adapts to any website size and structure. I have created an `llms.txt` for my website anyhow. I use a fixed LLM prompt to generate it from the internal links in `index.md`.
- iamacyborg 3mo agoGiving a markdown version of a page seems like an interesting choice instead of just embedding a schema marked up one
- vidarh 3mo agoEvery page on code.claude.com has a markdown version available by just appending ".md", and Claude Code knows about it. E.g: https://code.claude.com/docs/en/overview https://code.claude.com/docs/en/overview and https://code.claude.com/docs/en/overview.md https://code.claude.com/docs/en/overview.md
- deleted 3mo ago[deleted]
- 9dev 3mo agoAfter some consideration, I also applied this convention to every site I build - including content negotiation: Clients can either send an Accept header with their preference, or append an explicit extension (.md|.markdown for Markdown, .json for JSON API responses, or .html for the human HTML page). Together with the content negotiation part, it feels very much like HTTP was intended to work - especially the fact that API clients, AI agents, and humans all use the same URLs, but get the content in the shape they need.
- vidarh 3mo agoI've done this off and on for various sites over the years too, and probably should be more consistent about it. A number of sites do or used to do some variation of this, and I wish it was more widespread. E.g. Reddit will serve up a json version of a sub-dreddit if you do /r/subreddit.json
- mceachen 3mo agoHere's how to do it with more recent versions of Hugo: https://photostructure.com/coding/hugo-markdown-output/ https://photostructure.com/coding/hugo-markdown-output/ (It includes the grandparent's head link suggestion, but it's not just "change .html to .md" because I'm old skool and as a wee nerd was told that URLs ending in .html or .php or whatever we're frowned on, so the above link's markdown is available by appending /index.md )
- chrisweekly 3mo ago
- pfannl 3mo agoTo be fair, "not adopted by any major AI player" is probably the most web-standard-compliant phase of a new web standard.