7 ms·
Use the Accept Header to Serve Markdown Instead of HTML to LLMs
- skeptrune 1y agoThere was a lot of conversation about this on X over the last couple days and the `Accept` request header including "text/markdown, text/plain" has emerged as kind of a new standard for AI agents requesting content such that they don't burn unnecessary inference compute processing HTML attributes and CSS. - https://x.com/bunjavascript/status/1971934734940098971 https://x.com/bunjavascript/status/1971934734940098971 - https://x.com/thdxr/status/1972421466953273392 https://x.com/thdxr/status/1972421466953273392 - https://x.com/mintlify/status/1972315377599447390 https://x.com/mintlify/status/1972315377599447390
- hahnbee 1y agokeep us posted on how this change impacts your GEO!
- deleted 1y ago[deleted]
- burcs 1y agoReally cool idea Humans get HTML, bots get markdown. Two tiny tweaks I’d make... Send Vary: Accept so caches don’t mix Markdown and HTML. Expose it with a Link: …; rel="alternate"; type="text/markdown" so it’s easy to discover.
- yawaramin 1y agoThis person hypermedias
- Rohansi 1y agoWould be nice for humans to get the markdown version too. Once it's rendered you get a clean page.
- captn3m0 1y agoI’ve been asking for browser-native markdown support for years now. A clean web is not that far, if browsers support more than just HTML.
- lelanthran 1y ago> I’ve been asking for browser-native markdown support for years now. A clean web is not that far, if browsers support more than just HTML. You can always do the markdown -> DOM conversion on the client. Sure, there's a bit of latency there, but it means easier deployment (no build step involving pandoc or similar). Browser-native markdown support would be better though; you'd get ability to do proper contenteditable divs with bold, italic, etc done via markdown
- captn3m0 1y agoTo get broad support from the server side, you’ll need to showcase high browser support. We need Wordpress and Wikipedia and Ghost to support this, and that won’t happen without native browser support.
- lelanthran 1y ago> We need Wordpress and Wikipedia and Ghost to support this, and that won’t happen without native browser support. It can. Unlikely but possible. A good first step would be to have a well-written web component to be used like this: `<markdown>...</markdown>`, with no support at all for a build-step. The .js file implementing this should be included directly in the `<head>`. If that gets traction (unlikely, but possible) then the standards would sooner or later introduce a tag native to the browser that does the same thing.
- piquan 1y ago[dead]
- xigoi 1y ago
- foxfired 1y agoI think there is a problem of incentive here. When we made our websites Search Engine Optimized, the incentive was for google to understand our content, and bring traffic our way. When you make your content optimized for LLM, it only improves their product, and you get nothing in return.
- skeptrune 1y agoThis isn't true. ChatGPT and Gemini link to sites in a similar way to how search engines have always done it. You can see the traffic show up in ahrefs or semrush.
- nozzlegear 1y agoI had a call with a new user for a SaaS product that I sell recently. During the call he mentioned that he found it by typing what he was looking for into Gemini, and it recommended my app. I don't do anything special for llms, and the public-facing part of the website has been neglected for longer than I like to admit, so I was delighted. I had never considered that AI could send new users to me rather than pull them away. It felt like I'd hacked the system somehow, skipped through all the SEO best practices of yesteryear and had this benevolent bullshit machine bestow a new user on me at the cost of nothing.
- gl-prod 1y agoHow many users do actually visit these links?
- anabis 1y agoOpenAI cookbook says LLMs understand XML better than Markdown text, so maybe that also? Although, it should be more specified and structured, but not HTML.
- onion2k 1y agoOpenAI cookbook says LLMs understand XML better than Markdown text. Yes, for prompts. Given how little XML is out on the public internet it'd be surprising if it also applies to data ingestion from web scraping functions. It'd be odd if Markdown works better than HTML to be honest, but maybe Markdown also changes the content being served e.g. there's no menu, header, or footer sent with the body content.
- Kimitri 1y agoThe concept is called content negotiation. We used to do this when we wanted to serve our content as XHTML to clients preferring that over HTML. It's nice to see it return as I always thought it was quite cool.
- klodolph 1y agoI don’t understand why the agents requesting HTML can’t extract text from HTML themselves. You don’t have to feed the entire HTML document to your LLM. If that’s wasteful, why not have a little bit of glue that does some conversion?
- skeptrune 1y agoIt's always better for the agent to have fewer tools and this approach means you get to avoid adding a "convert HTML to markdown" one which improves efficiency. Also, I doubt most large-scale scrapers are running in agent loops with tool calls, so this is probably necessary for those at a minimum.
- klodolph 1y agoThis does not make any sense to me. Can you elaborate on this? It seems “obvious” to me that if you have a tool which can request a web page, you can make it so that this tool extracts the main content from the page’s HTML. Maybe there is something I’m missing here that makes this more difficult for LLMs, because before we had LLMs, this was considered an easy problem. It is surprising to me that the addition of LLMs has made this previously easy, efficient solution somehow unviable or inefficient. I think we should also assume here that the web site is designed to be scraped this way—if you don’t, then “Accept: text/markdown” won’t work.
- hahnbee 1y agoIf you have a website and you're optimizing it for GEO, you can't assume that the agents are going to have the glue. So as the person maintaining the website you implement as much of the glue as possible.
- klodolph 1y agoThat sounds completely backwards. It seems, again, obvious to me that it would be easier to add HTML->markdown converters to agents, given that there are orders of magnitude more websites out there compared to agent. If your agent sucks so bad that it isn’t capable of consuming HTML without tokenizing the whole damn thing, wouldn’t you just use an agent that isn’t such a mess? This whole thing kinda sounds crazy inefficient to me.
- NathanFlurry 1y agoWe’re doing this on https://rivet.dev https://rivet.dev now. I did not realize how much context bloat we had since we were using Tailwind.
- skeptrune 1y agoIt is crazy how badly Tailwind bloats HTML. Tradeoffs!
- stebalien 1y agoOr one can just use semantic HTML; it's easy enough to convert semantic HTML into markdown with a tool like pandoc. That would also help screen readers, browser "reader modes", text-based web browsers, etc.
- jauntywundrkind 1y agoMaybe adopt the existing Gemini Protocol instead? It's already a nice very simple markdown-like. https://toffelblog.xyz/blog/gemini-overview/ https://toffelblog.xyz/blog/gemini-overview/ https://news.ycombinator.com/item?id=23730408 https://news.ycombinator.com/item?id=23730408 https://gemini.circumlunar.space/ https://gemini.circumlunar.space/ https://news.ycombinator.com/item?id=23042424 https://news.ycombinator.com/item?id=23042424
- troyvit 1y agoFYI both the link to toffelblog and circumlunar.space are broken with ssl errors.