rohaga

6 ms·

1.
by rohaga 2mo ago
We have run it on DeepSWE, check out our blog post for details of how we leverage the intelligence of the smart models https://usetokenless.com/blog/building-tokenless/
2.
by rohaga 2mo ago
Since we fan out in parallel, by definition the latency is only at worst the latency of the slowest model. The model in between is fairly light, so it seems to not materially affect things. Deciding mid-turn also lets us resolve earlier, of
3.
by rohaga 2mo ago
Yeah this is a great observation--we have to collect some more data to understand new models as they come out. However, by having this composable architecture that separates "confidence prediction" and the actual "routing lay
4.
by rohaga 2mo ago
I agree that adjusting the reasoning effort to retain cache is a huge thing! But even doing that automatically is currently a challenge for people to figure out and do well, and costs mental energy when perhaps it doesn't need to. For
5.
by rohaga 2mo ago
It automatically works with multi-turn routing. You can try it yourself in Claude Code or Codex
6.
by rohaga 2mo ago
Thanks for the praise! We think the results are a work in progress. Routing is convincing for quality for the following reason. Take your favorite benchmark, and on each task, run both the top GPT and top Claude model, and call the task a s
7.
by rohaga 2mo ago
Well, you can estimate the confidence BEFORE you start the task, too. That way you can restrict your trajectory to just a few models. We also think there are tons of people working on "context management"--e.g. retrieval systems,
8.
by rohaga 2mo ago
Highly encourage you to read the blog post ( https://usetokenless.com/blog/building-tokenless ). Essentially, we estimate the confidence of a specific model failing or succeeding on a specific task using our own foundati
9.
by rohaga 2mo ago
Our research seems to indicate you can get a decent signal based on the first few reasoning trace summaries. We analyzed hundreds of billions of tokens to figure this out. Any open-weights model also gives you reasoning, which is very helpf
10.
by rohaga 2mo ago
If we cut off an unconfident/expensive model early in thinking, then we don't have to pay for additional thinking cost at all. For many tasks, this is quite a large saving. That being said, whatever models we select to race at any
11.
by rohaga 2mo ago
> So this only switches models if the cache is cold Not exactly. It can also make sense if the cache is hot to switch models. For example, we might currently be on Deepseek and the next task is SO HARD that it doesn't make sense to
12.
Launch HN: Tokenless (YC S26) – Automatic model switching to save money(usetokenless.com)
71 points by rohaga 2mo ago|63 comments