danlenton

7 ms·

1.
by danlenton 4mo ago
I think the main benefit is improved speed and parallelism. Very similar to https://thinkingmachines.ai/blog/interaction-models/
2.
Unify (YC W23) Is Hiring LLM Engineers (Contributors)(ycombinator.com)
1 points by danlenton 2y ago
3.
by danlenton 2y ago
We just initialize a random latent vector for each model, and then jointly train each of these unique latent vectors :)
4.
by danlenton 2y ago
Currently the motivation is mainly speed. For the really easy ones like "hey, how's it going?" or "sorry I didn't hear you, can you repeat?" you can easily send to Llama3 etc. Ofc you could do some clever cachi
5.
by danlenton 2y ago
Interesting, do you have any hunch as to why this is? We've seen in more verticalized apps where the underlying model is hidden from the user (sales call agent, autopilot tool, support agent etc.) that trying to reach high quality on h
6.
by danlenton 2y ago
no down side
7.
by danlenton 2y ago
One use case is optimizing agentic systems, where a custom router [ https://youtu.be/9JYqNbIEac0 ] is trained end-to-end on the final task (rather than GPT4-as-a-judge). Both the intermediate prompts and the models used can t
8.
by danlenton 2y ago
Thanks for sharing, will get this fixed now!
9.
by danlenton 2y ago
If you do test it out, feel free to ping me with any questions!
10.
by danlenton 2y ago
Makes sense, however I would clarify that we don't need to make the final decision. If you're using the neural scoring function as an API, then you can just get predictions about how each model will likely perform on your prompt,
11.
by danlenton 2y ago
Thanks for weighing in. I'm sure for your setup right now, our router in it's current form would not be useful for you. This is the very first version, and the scope is therefore relatively limited. On our roadmap, we plan to supp
12.
by danlenton 2y ago
duly noted!
13.
by danlenton 2y ago
however, as janekm says, we can't charge just based on cost savings. We would need the router points to be sufficiently compelling wrt quality, speed and cost (including our own margins) that users still sometimes opt for these router
14.
by danlenton 2y ago
Yeah that's a great point, something we'll keep in mind as we work out the final business model. Thanks!
15.
by danlenton 2y ago
Thanks! Ipsos is also a great analogous example, I hadn't thought of that.
16.
by danlenton 2y ago
Makes sense, thanks a lot for the feedback. We're pretty confident that future versions of our router will provide sufficient value where we can take margins here, we therefore don't expect the need to start charging for Single-si
17.
by danlenton 2y ago
It's on the roadmap! Hopefully will be added next week
18.
by danlenton 2y ago
Yes the benchmarks are ongoing, we continually plot the speed and cost across time in our runtime benchmarks [ https://unify.ai/benchmarks ], and we use this live data when plotted the quality scatter graphs [ https:/&#x
19.
by danlenton 2y ago
Currently, we simply use GPT4-as-a-judge, with a general system prompt we've written which is task agnostic. This is then used to train the neural scoring function, which predicts quality ahead-of-time. However, it's on our roadma
20.
by danlenton 2y ago
Sure! Basically traditional MoE has several linear layers, and the network learns to route down those paths, based on the training loss (similar to how CNNs learn through max-pooling, which is also non-differentiable). However, MoEs have be
21.
by danlenton 2y ago
So the neural scoring introduces ~20ms latency, but this only impacts time-to-first-token (not inter-token-latency). When using our public endpoints there is an additional ~150ms latency, but you can deploy the router on-prem in your own cl
22.
by danlenton 2y ago
lol well sure, that too
23.
by danlenton 2y ago
Great question! Nope they're not stuck with the simple LLM after the first question. Chatbots simply aggregate all prior conversation history in the input prompt, and make a new LLM call. The router treats each LLM call seperately, and
24.
by danlenton 2y ago
absolutely! You're also welome to schedule a call: https://calendly.com/unify-chat/general Best email would be: daniel.lenton@unify.ai Cheers!
25.
by danlenton 2y ago
actually we don't support local deployment quite yet, it's all run in our server. However, fully local deployment is on the roadmap, stay tuned!
26.
by danlenton 2y ago
aha good question, well the neural scoring function doesn't "know" that it is making a routing decision, we just predict which LLM would give the highest performing output on the given prompt, based on LLMs-as-a-judge at trai
27.
by danlenton 2y ago
Yeah I agree, routing all the time is overkill for some applications, however for others it really is necessary in order to reach the necessary speed and performance for users of your real-time app. For those who don't want to always r
28.
by danlenton 2y ago
We use this data to improve the base router by default. It's fully anonymized, and you can opt out.
29.
by danlenton 2y ago
Yep it's the same idea. We also see improvements in quality on some benchmarks beyond any specific model. This is common especially common when training a custom router on your own prompt dataset: https://youtu.be/9JYqN
30.
by danlenton 2y ago
I agree this is an interesting direction, I think this is on the roadmap for DSPy [ https://github.com/stanfordnlp/dspy ], but right now they mainly focus on optimizing the in-context examples.

More ›