5 ms·
Show HN: VLLM with JSON Guided Generation
Our project, Outlines, now offers guided/constrained generation (e.g. according to a JSON schema) via the VLLM library.
My colleague, Rémi, created some patches that allow one to pass vLLM a JSON schema along with the prompt, which dramatically simplifies deployment of JSON-guided generation. He also added a new `serve` interface that puts it all together and makes serving such models a 2-3 line process.
Check it out and tell us what you think!
- jilijeanlouis 3y agothis is really cool probably the most wanted feature in LLM open source. It would be nice to have regex support too? do you plan to support regex?
- remilouf 3y agoWe already support regex-guided generation in the library, and could easily make an API to serve this as well if that's a feature people want!
- jilijeanlouis 3y agoI need it yes. that would be amazing tbh.