9 ms·
Show HN: Optimize and serve models with Fable quality at half the cost
Hi HN, we built world-model-optimizer, an open source tool to continually improve a specialized model for an agent.
It does this by simulating production tool responses through text world modeling (similar to QwenAgentWorld, summary here https://x.com/silennai/status/2073887455884058814 https://x.com/silennai/status/2073887455884058814).
We can then use this to train a router for frontier, OS, and local models (use defaults or pick which ones to optimize against).
wmo ingests agent traces, builds the simulation, embeds the traces, runs different models you choose against the simulation scenarios, and then uses a KNN for model selection (similar to https://arxiv.org/abs/2505.19797 https://arxiv.org/abs/2505.19797).
- Cache aware: cache is taken into account for the effective price in routing.
- Confidence gated: we don't deviate from the best fit model when paired evidence over retrieved neighbors is below 0.5 standard errors or on queries unlike anything in the fit set.
- Optimize for cost or quality: train a balanced, cost max, or quality max router.
Usage
`wmo build` creates the simulation (or add your own benchmark)
`wmo optimize` tunes the router
`wmo serve` starts the server and can run everything fully locally. The simulation and router can update over time as more agent traces are gathered and new models are added.
Router results vs Fable
- RouterBench: -66.5% cost, -1.7% performance, -24.7% latency p50. 77.5% of traffic to Sonnet 5, 16.1% Fable 5.
- TauBench: -44.5% cost, +6.3% performance, -20% latency. 83% to Opus 5, 17% to Kimi-K2.6 (over K3).
- Terminal Bench 2: -64% cost, +8% performance, -50.6% latency. Sonnet 5 is fully along the pareto front. Training a specialized router per task isn't cheap. In sparse data regimes the value can be "here's the best model".
We're working on sample effiient continual learning for agent specific models at experientiallabs.ai"
- yiyingzhang 2mo agoCool idea! How do you guarantee privacy?
- SilenN 2mo agoIt's open source! We do have a platform we'll be launching as well to manage training + serving for you which will require more diligent privacy guarantees.
- digitaltrees 2mo agoCool project
- SilenN 2mo agoThanks :)
- rglover 2mo agoExcited to play with this.
- SilenN 2mo agoLet me know if you have any questions!
- jack_pp 2mo agoNot sure I get it. The model you're improving is local? If so how do you even calculate cost compared to an API
- SilenN 2mo agoOpen source models. wmo routes requests between frontier models and open source models that continuously train using Tinker. As the smaller models improve, more traffic gets routed to them. Calculating cost is just tokens in/out.
- handfuloflight 2mo agoWhat are the costs to train and use the Tinker models?
- SilenN 2mo agoExpensive, in the thousands. We have our own infra in house and are working on bringing these costs down
- handfuloflight 2mo agoBut in the thousands can pencil out if you're saying tens of thousands over calling the frontier models, no?
- SilenN 2mo agoExactly
- adrianco 2mo agoLocal models need to be tuned to work well so this looks useful. Seems to be for general purpose model serving. I’ve been using https://github.com/adrianco/retort https://github.com/adrianco/retort to run experiments for coding models across 13 different programming languages to see which frontier and local models work.
- SilenN 2mo agoThat's cool, thanks for sharing!
- Art9681 2mo agoThe absolute best way to prove this works is by releasing a model that was fine-tuned with this method and then showing benchmarks depicting the improvement delta between the base model and the fine tuned one. The work is not done. Then release it to the masses and wait a few days for the actual real world anecdotes. Until then, this is noise.
- Reubend 2mo agoYeah, this is just slop. No benchmarks, no concrete case studies, just some vibecoded "platform" to finetune models on your own traces. Which is an idea that has some value, but also some weaknesses. And this implementation of it isn't forthcoming with that concept. You have to really dig in to understand what they're even talking about.
- SilenN 2mo agoHappy to answer any qs.
- irishcoffee 2mo agoBenchmarks are the ultimate consolidation of halnons razor.
- teravor 2mo ago[flagged]
- dang 2mo ago"Please don't post shallow dismissals, especially of other people's work. A good critical comment teaches us something." https://news.ycombinator.com/newsguidelines.html https://news.ycombinator.com/newsguidelines.html
- SilenN 2mo agoValid criticism. Happy to answer any qs. We're still working on solidfying results.
- surround 2mo agoThe title is misleading. This is model routing, not distillation.
- SilenN 2mo agoFixed formatting which will help with readability. We do routing, distillation, and token compaction.
- renezander030 2mo ago[flagged]
- anshad2u 2mo agoInteresting approach. What does the cold-start phase look like for a new agent? How many traces or runs do you typically need before the router has enough signal to safely offload tasks from the frontier model??
- SilenN 2mo agoTechnically 0 because a) it ingests your already existing traces and does an initial training run b) in the app we'll have pre-trained routers you can start with that will then learn over time
- h2aichat 2mo agoProbably a silly question, but ...: if it's optimizing itself purely inside a simulator, what keeps it from just exploiting the simulator's glitches to look good on paper, while not being so good when it hits the real world?
- kfallah15 2mo agoGreat question. You need the simulator to be realistic enough that the optimizer cannot reward hack, but it does not need to be perfectly realistic.
- SilenN 2mo ago^this There's two ways to functionally measure this, reconstruction fidelity (which we're able to get to 0.7 - 0.95), and downstream performance (which agrees on the best and worst models, but can sometimes disagrees on the middle of the pack routing).
- deleted 2mo ago[deleted]
- diwesh871 2mo ago[flagged]