6 ms·
Sampling and structured outputs in LLMs
- SamLeBarbare 1y agoThis post dives into that "black magic" layer, especially in the context of emerging thinking models and tools like Ollama or GPT-OSS. It’s a thoughtful look at why sampling, formatting, and standardization are not just implementation details, but core to the future of working with LLMs.
- electroglyph 1y agoI don't know if you're purposely trying to be funny, but this is obnoxious, lol
- k__ 1y agoSounds like brute force to me.
- _1tem 1y agoHmm, so if structured output affects the quality of the response maybe it's better to convert the output to a structured format as a post-processing step?
- NitpickLawyer 1y agoIt's a tradeoff between getting "good enough" performance w/ guided/constrained generation and using 2x calls to do the same task. Sometimes it works, sometimes it's better to have a separate model. One good case of 2 calls is the "code merging" thing, where you "chat" with a model giving it a source file + some instruction, and if it replies with something like ... //unchanged code here ... some new code ... //the rest stays the same, then you can use a code merging model to apply the changes. But that's become somewhat obsolete by the new "agentic" capabilities where models learn how to diff files directly.
- BoredPositron 1y agoHaiku is my favorite model for the second pass. It's small cheap and usually gets it right. If I see hallucinations they are mostly from the base model in the first pass.
- hedgehog 1y agoDepending on the task you can often get it in about one request on average. Ask for the output in Markdown with reasoning up front and the structured output in a code block at the end, then extract and parse that bit in code.
- ninadpathak 1y ago[dead]
- l5870uoo9y 1y agoAfter endlessly tweaking the SQL generators[1] that I am working on, I would recommend setting a "reasoning" output string to activate step by step thinking and better responses. Even better if you can add output "reasoning strings" more relevant to the specific task you are trying to solve. [1]: https://app.sqlai.ai https://app.sqlai.ai
- thrance 1y agoIt's still baffling to me that the various API providers don't let us upload our custom grammars. It would enable so many use cases, like HTML generation for example, at essentially no cost on their part.
- bubblyworld 1y agoWouldn't that have implications for inference batching, since you would have to track state and apply a different mask for each sequence in the batch? If so, I think it would directly affect utilisation and hence costs. But I could be talking out of my ass here.
- esafak 1y agoWhen you say custom grammar, do you mean something other than a JSON schema, because they support that?
- thrance 1y agoI mean, most don't? I know you can provide a pseudo-EBNF grammar to llama.cpp but, for example, none of Anthropic, Azure, Bedrock, Mistral or Gemini allow us the same.
- barrkel 1y agoUsing grammar constrained output in llama.cpp - which has been available for ages and I think is a different implementation to the one described here - does slow down generation quite a bit. I expect it has a naive implementation. As to why providers don't give you a nice API, maybe it's hard to implement efficiently. It's not too bad if inference is happening token by token and reverting to the CPU every time, but I understand high performance LLM inference uses speculative decoding, with a smaller model guessing multiple tokens in advance and the main model doing verification. Doing grammar constraints across multiple tokens is tougher, there's an exponential number of states that need precomputing. So you'd need to think about putting the parser automaton onto the GPU/TPU and use it during inference without needing to stall a pipeline by going back CPU. And then you start thinking about how big that automaton is going to be. How many states, pushdown stack. You're basically taking code from the API call and running it on your hardware. There's dragons here, around fair use, denial of service etc.
- frotaur 1y agoWhen doing structured sampling, why is the token sampled, checked against the grammar, and resampled if it's wrong by applying the mask ? Why wouldn't we apply the mask immediately for the first sampling? Is this an optimization somehow, is masking expensive?
- 2THFairy 1y agoImplementation preference. > is masking expensive? It's not expensive per-se; A single element-wise multiplication of the output vector. The real "expense" is that you need to prepare masks for every element of your grammar as they are expensive to recompute as needed; LLM tokens do not cleanly map onto elements of your grammar. (Consider JSON: LLM tokens often combine various special characters such as curly braces, colons, and quotes.) This isn't that hard to compute, it's just more work to implement.
- FlyingLawnmower 1y agoIf you can screen tokens against your grammar fast enough, you can build a bitmask over the entire token vocabulary and apply it right before sampling. As vocabulary sizes grow, this gets more complex to do in real time, but we (and other libraries) have found several optimizations to do this extremely quickly (eg for guidance, we detail some optimizations here https://github.com/guidance-ai/llguidance/blob/main/docs/optimizations.md https://github.com/guidance-ai/llguidance/blob/main/docs/opt...). Other libraries work by essentially pre-computing all the masks for all possible generations, but of course you're restricted to working with simple grammars in this case (like a subset of regular expressions)
- ninadpathak 1y ago[dead]
- parthsareen 1y agoHey! I'm the author of the post. We haven't optimized sampling yet so it's running linearly on the CPU. A lot of SOTA work either does this while the model is running the forward pass or does the masking on the GPU. The greedy accept is so that the mask doesn't need to be computed. Planning to make this more efficient from either ends.
- amelius 1y agoThis constrains the output of the LLM to some grammar. However, why not use a grammar that does not have invalid sentences, and from there convert to any grammar that you want?
- cyptus 1y agoWhat if the converted version is not in the wanted syntax?
- NitpickLawyer 1y agoConstrained generation guarantees syntax. It does not guarantee semantic correctness tho. Imagine you want a json object with "hp" and "damage". If you use a grammar, the model will be forced to output a json object with those two values. But it's not guaranteed to get sensible values. With a 2nd pass you basically "condition" it on the text right above, hoping to get better semantic understanding.
- lyu07282 1y agoI'm pretty sure the grammar is generated from the Json schema, it doesn't just constrain json syntax, it constraints on the schema (including enums and such). The schema is also given to the model (at least in openai) you can put instructions in the json schema as well that will be taken into account.
- NitpickLawyer 1y agoPerhaps I worded that poorly. What I mean by semantic correctness is that the model could output nonsensical values for some things. Say in a game, "normal" health is ~100hp and the model creates a wizard with 50hp but then a mouse with 10000hp. So you're guaranteed to get a parsable json object (syntactically correct) but what the values are in that json is not guaranteed to make sense in the given context.
- 1y ago
- CuriouslyC 1y agoJust wait till people realize that if you have agents speak in structured output rather than chatting with you, your observability and ability to finely program your agent goes through the roof.
- electroglyph 1y agoIf the current position in the structure only has one possibility (like a comma, bracket, etc.) do you just force that as the next token and continue?
- maccam912 1y agoI don't think so, because multiple tokens might match. If it needs a comma as the next character, but you have tokens for `, "blah` and `, "foo` you still want to leave those on the table.
- FlyingLawnmower 1y agoWe do enable forcing these sequences of tokens in guidance, and find that it significantly speeds up structured generation. There are tricky alignment issues to make sure you pick the right sequence of tokens, but you can often proxy this well by using the model's native tokenizer. Some details here in an old blog: https://guidance.readthedocs.io/en/latest/example_notebooks/tutorials/token_healing.html https://guidance.readthedocs.io/en/latest/example_notebooks/...
- ninadpathak 1y ago[dead]
- iandanforth 1y agoThis is a great writeup! There was a period where reliable structured output was a significant differentiator and was the 'secret sauce' behind some companies success. A NL->SQL company I am familiar with comes to mind. Nice to see this both public and supported by a growing ecosystem of libraries. One statement surprised me was that the author thinks "models over time will just be able to output JSON perfectly without the need for constraining over time." I'm not sure how this conclusion was reached. "Perfectly" is a bar that probabilistic sampling cannot meet.
- ninadpathak 1y ago[dead]
- joatmon-snoo 1y agoWe’ve had a lot of success implementing schema-aligned parsing in BAML, a DSL that we’ve built to simplify this problem. We actually don’t like constrained generation as approach - among other issues it limits your ability to use reasoning - and instead the technique we’re using is algorithm-driven error-tolerant output parsing. https://boundaryml.com/ https://boundaryml.com/
- maxdo 1y agoLove your work , thanks ! , 12 factor agent implementation uses your tools too.
- parthsareen 1y agoThank you! Maybe not "perfect" but near-perfect is something we can expect. Models like the Osmosis structure which just structure data inspired some of that thinking (https://ollama.com/Osmosis/Osmosis-Structure-0.6B https://ollama.com/Osmosis/Osmosis-Structure-0.6B). Historically, JSON generation has been a latent capability of a model rather than a trained one, but that seems to be changing. gpt-oss was particularly trained for this type of behavior and so the token probabilities are heavily skewed to conform to JSON. Will be interesting to see the next batch of models!
- FlyingLawnmower 1y agoI spent a couple years building a high performance, expressive library for structured outputs in LLMs. Our library is used by OpenAI for structured outputs on the hosted API. Happy to answer questions on how this works: User friendly library that connects to lots of OSS model serving backends: https://github.com/guidance-ai/guidance/ https://github.com/guidance-ai/guidance/ Core Rust library written for high performance mask computation (written mostly by my collaborator @mmoskal): http://github.com/guidance-ai/llguidance http://github.com/guidance-ai/llguidance
- ninadpathak 1y ago[dead]
- FlyingLawnmower 1y agoThanks :) Great question re: adoption...it's definitely dominated by JSON. Most API providers have standardized on JSON outputs, so application teams have started building shims that map other formats to JSON and back. Similarly, with models heavily being post-trained to generate "good" JSON, I think there's a better model-constraint alignment story with JSON than most arbitrary grammars. That said, internally, we experiment quite a lot with custom grammars all across the stack. It's more complicated to write a grammar than a JSON schema (though LMs are very good at grammar writing now) and more error prone to debug, but it can help significantly in certain cases (e.g. having models write custom DSLs not commonly found on the internet, at various parts of a model training pipeline, etc. etc.). I'm hoping that with the right tooling around it, the broader community will start nudging beyond JSON. To that end, the python guidance library is really an attempt to make writing grammars more friendly to a python programmer. More to be done here of course!
- btown 1y agoThe LLGuidance paper is highly recommended reading for everyone interested in this! https://guidance-ai.github.io/llguidance/llg-go-brrr https://guidance-ai.github.io/llguidance/llg-go-brrr TL;DR instead of just getting a token and seeing if it would be accepted by the parser, you can actually zero-out probabilities for all invalid tokens, and do the computation for this in parallel at effectively zero cost: > Here, compute_mask() can run on the CPU during the time it would be normally just waiting for the GPU to finish. The line prob[~mask] = 0.0 would normally be fused into the softmax kernel in the last stage of the LLM, with negligible overhead. Therefore, as long as the compute_mask() function completes faster than the LLM forward pass and parser.consume() is negligible (typically follows from compute_mask() speed), the constrained generation will be as fast as the unconstrained one. I'm curious - have there been any research/conversations about pushing masking even earlier in the pipeline? In theory, there's a fair amount of compute that goes into computing the probability of tokens that will end up being masked away anyways.
- minimaxir 1y agoGoogle's Gemini API is a bit odd with structured outputs. If you specify an Application/JSON response mimetype, it will reliably respond with a consistent JSON output without any prompt engineering shenanigans. For my workflows, this setting plus providing a JSON Schema in the system prompt works even with complex schema. The Gemini API has a canonical implementation of structured outputs where you can instead pass the JSON schema as a separate parameter to control the grammar more closely. However, this setting will reorder the JSON schema fields to be alphabetical beforehand, which is especially not desired behavior as the order of JSON fields in a schema is often very deliberate to control generation.
- madethemcry 1y agoThat was as great reading, thank you. I've a related observation. In my experience the amount of hallucinated urls with structured output (think of a field `url` or `link`) is pretty high. Especially compared to the alternative approach, where you let the llm generate text and then use a second llm to convert the text into the desired structured format. With structured output, it's like the llm is forced to answer in a very specific way. So if there is no url for the given field, it makes up the url. Here a related quote from the article: > Structured outputs builds on top of sampling by constraining the model's output to a specific format.
- miki123211 1y agoWhat I've found is that it is very important to make structured outputs as easy for the LLM as possible. This means making your schemas LLM-friendly instead of programmer-friendly. E.g. if the LLM hallucinates non-existing URLs, you may add a boolean "contains_url" field to your entity's JSON schema, placing it before the URL field itself. This way, the URL extraction is split into two simpler steps, checking if the URL is there and actually extracting it. If the URL is missing, the `"contains_url": false` field in the context will strongly urge the LLM to output an empty string there. This also comes up with quantities a lot. Imagine you're trying to sort job adverts by salary ranges, which you extract via LLm. . These may be expressed as monthly instead of annual (common in some countries), in different currencies, pre / post tax etc. Instead of having an `annual_pretax_salary_usd` field, which is what you actually want, but which the LLM is extremely ill-equipped to generate, have a detailed schema like `type: monthly|yearly, currency:str, low:float, high:float, tax: pre_tax|post_tax`. That schema is much easier for an LLM to generate, and you can then convert it to a single number via straight code.
- lubujackson 1y agoAwesome insight, thanks for this!
- hansvm 1y agoThat's definitely possible. As you know, (most current) LLMs build text autoregressively. This allows them to generate text with _exactly_ the same distribution as the training data. When you constrain LLM output at each token, that gives a completely different distribution from letting the LLM generate a full output and then doing something with that (trying again, returning an error, post-processing, etc). E.g.: Suppose the LLM has a training set of (aa, ab, ab, ba), noting that "ab" appears twice. Suppose your valid grammar is the set (ab, ba). Then your output distributions are: Baseline: {invalid: 25%, ab: 50%, ba: 25%} Constrained: {invalid: 0%, ab: 75%, ba: 25%} Note that _all_ the previously invalid outputs were dumped into the "ab" bucket, skewing the ratio between "ab" and "ba". That skew may or may not be desirable, but assuming the training process was any good it's likely undesirable. You've observed it in URLs, but I see it in JSON output as well. LLMs like to truncate long strings from time to time, but when they do they're more likely to provide invalid JSON (adding an ellipsis at the end of the fragment and doing nothing else). If that truncation starts to happen in a constrained environment, a period is a valid character in a long string, and eventually the grammar constraint will force a closing quote to appear. The result is still garbage, but instead of a detectable parse failure you have an undetectable corrupt field.
- hellovai 1y agoHave you tried techniques that don’t require modifying the LLM and the sampling strategy for structure outputs? For example, schema aligned passing, where you build error tolerance into the parser instead of coercing to a grammar. https://boundaryml.com/blog/schema-aligned-parsing https://boundaryml.com/blog/schema-aligned-parsing
- Jonovono 1y agoI love BAML. Surprised it’s not more popular. I can get structured outputs on any model even ones that don’t support json schema outputs etc
- hedgehog 1y agoIt looks really slick, for us the reason we haven't adopted yet is it brings more tooling and configuration that overlaps with our existing system for prompt templates, schema definitions, etc. In the component where we couldn't rely on OpenAI structured outputs we experimented with TOML-formatted output, that ended up being reliable enough to solve the problem across many models without any new dependencies. I do think we'll revisit at some point as Boundary also provides incremental parsing of streaming outputs and may allow some cost optimization that is not easy right now.
- benob 1y agoThese techniques are limited to structures that can be checked with bounded history or bounded memory (that can be checked with a grammar or FSA). What about more complex structures that don't factor easily?
- deleted 1y ago[deleted]
- visarga 1y agoI was hoping to find some insights about why performance drops when using actual structured outputs. It's been a known problem. For example this paper "Let Me Speak Freely? A Study on the Impact of Format Restrictions on Performance of Large Language Models" says: > Surprisingly, we observe a significant decline in LLMs’ reasoning abilities under format restrictions. Furthermore, we find that stricter format constraints generally lead to greater performance degradation in reasoning tasks. https://arxiv.org/abs/2408.02442v1 https://arxiv.org/abs/2408.02442v1
- roadside_picnic 1y agoThat paper had some serious methodological issues and the results have been shown to be misunderstood/incorrect in the majority of cases. In fact, in many cases structured outputs have shown to improve the quality of the results from an LLM (at least in terms of evaluation performance). The team at behind the Outlines library released a response the covers the issues in details and provides more information about structured outputs [0]. 0. https://blog.dottxt.ai/say-what-you-mean.html https://blog.dottxt.ai/say-what-you-mean.html
- heckintime 1y agoI've found that writing a very simple DSL that resembles human speech and an interpreter that can output JSON is very effective. Human 4x1200 with 30 second rest AI DSL output Repeat 4 times: - Run 1200 meters - Rest 30 seconds I hand wrote a recursive descent parser in Python to process DSL. Human speech to DSL is pretty effective with a simple prompt and some examples. I created a tool that can program Garmin & Apple Watches for interval training based on what I wrote above. https://speedystride.com https://speedystride.com Looking for beta testers- please give it a try :)
- constantinum 1y agoAnother common stack that is commonly use is Langchain + Pydantic https://unstract.com/blog/comparing-approaches-for-using-llms-for-structured-data-extraction-from-pdfs/ https://unstract.com/blog/comparing-approaches-for-using-llm...
- parthsareen 1y agoThanks for posting! Didn't expect this to get picked up – it was a bit of a draft haha. Happy to answer questions around structured outputs :)
- infecto 1y agoMy takeaway still today is that structured output is a two part process. If you require any heavy lifting on the LLM side, introducing structured output is going to cause reduced quality.
- EnnEmmEss 1y agoThis is off-tangent but I find it a bit odd that the blog uses a URL fragment to load different articles when it's usually used to navigate within a page. A consequence of this seems to be that clicking the link to a different article leaves you at the bottom of the page even though the article itself has changed. This seems to be using JS to fetch the markdown and then render it but I do feel that it may be better off to simply pre-convert the markdown as part of the deployment process and serve the static page.
- parthsareen 1y agoThat's a great idea. Going to try this next :)
- EnnEmmEss 1y agoHappy to help :)