5 ms·
As someone also building constrained decoders against JSON [1], I was hopeful to see the same but I note the following from their documentation: The model ca
by newhouseb 3y ago
As someone also building constrained decoders against JSON [1], I was hopeful to see the same but I note the following from their documentation:
The model can choose to call a function; if so, the content will be a stringified JSON object adhering to your custom schema (note: the model may generate invalid JSON or hallucinate parameters).
So sadly, it is just fine tuning. There's no hard biasing applied :(. You were so close, but so far OpenAI!
[1] https://github.com/newhouseb/clownfish https://github.com/newhouseb/clownfish
[2] https://platform.openai.com/docs/guides/gpt/function-calling https://platform.openai.com/docs/guides/gpt/function-calling
- civilitty 3y agoOr there’s a trade off between more complex schemas and logit bias going off the rails since there’s probably little to no backtracking.
- newhouseb 3y agoGood point. Backtracking is certainly possible but it is probably tricky to parallelize at scale if you're trying to coalesce and slam through a bunch of concurrent (unrelated) requests with minimal pre-emption.
- jumploops 3y agoThey may have just fine-tuned 3.5 to respond with valid JSON more times than not. Building magic functions[0] I ran into many examples where JSONSchema broke for gpt-3.5-turbo but worked well for gpt-4. [0] https://github.com/jumploops/magic https://github.com/jumploops/magic