5 ms·
Having some success while testing this model out as a replacement for GPT-5 nano in source code security review. Running on RTX 3090 (24 GB VRAM) via vLLM. It's
by noperator 3mo ago
Having some success while testing this model out as a replacement for GPT-5 nano in source code security review. Running on RTX 3090 (24 GB VRAM) via vLLM. It's not great on structured output (as noted in the model card) but I'm working around that in my harness.
- dummydummy1234 3mo agoCan't you just force it to do structured output via constrained generation?
- noperator 3mo agoYes, I did end up figuring out a clean way to allow normal reasoning inside <think> and then force JSON _after_ the closing </think>. Example here: https://gist.github.com/noperator/6c711ab19027ea8056442df839f2d7e6 https://gist.github.com/noperator/6c711ab19027ea8056442df839...
- nickalaso 3mo agoThanks for that, and credit to you for the idea, I quickly vibe coded (heh) a working minimal tool calling harness thats allows it to make multiple tool calls each turn. Currently its working pretty damn well all things considered: https://github.com/NickalasLight/VibeHarness.git https://github.com/NickalasLight/VibeHarness.git
- hypfer 3mo ago> but I'm working around that in my harness. How?
- uberex 3mo agoMaybe limiting logits to what is syntactically correct? E.g. {"hello" has to be followed by whitespace or colon. Any other logits get dropped.