5 ms·

At 16k tokens/s why bother routing? We're talking about multiple orders of magnitude faster and cheaper execution. Abundance supports different strategies. One

At 16k tokens/s why bother routing? We're talking about multiple orders of magnitude faster and cheaper execution. Abundance supports different strategies. One approach: Set a deadline for a response, send the turn to every AI that could possibly answer, and when the deadline arrives, cancel any request that hasn't yet completed. You know a priori which models have the highest quality in aggregate. Pick that one.

IanCal 7mo ago
The best coding model won’t be the best roleplay one which won’t be the best at tool use. It depends what you want to do in order to pick the best model.
- PhunkyPhil 7mo ago
  I'm not saying you're wrong, but why is this the case? I'm out of the loop on training LLMs, but to me it's just pure data input. Are they choosing to include more code rather than, say fiction books?
  - refulgentis 7mo ago
    I’ll go ahead and say they’re wrong (source: building and maintaining llm client with llama.cpp integrated & 40+ 3p models via http) I desperately want there to be differentiation. Reality has shown over and over again it doesn’t matter. Even if you do same query across X models and then some form of consensus, the improvements on benchmarks are marginal and UX is worse (more time, more expensive, final answer is muddied and bound by the quality of the best model)
    - SignalStackDev 7mo ago
      [dead]
    - IanCal 7mo ago
      Are you saying I’m wrong that some models are better for some tasks than others, but there isn’t a universally best model for all tasks?
    - stephenbez 7mo ago
      Thanks. Are there any links where I can learn more about this? I did some Googling and it appears that there are some examples where people say combining multiple models or multiple runs of the same models leads to improvements: https://www.sciencedirect.com/science/article/abs/pii/S0010482525010820 https://www.sciencedirect.com/science/article/abs/pii/S00104... https://arxiv.org/abs/2203.11171 https://arxiv.org/abs/2203.11171 But presumably people are less likely to publish a paper when an approach doesn’t work.
  - jmalicki 7mo ago
    There is the pre-training, where you passively read stuff from the web. From there you go to RL training, where humans are grading model responses, or the AI is writing code to try to pass tests and learning how to get the tests to pass, etc. The RL phase is pretty important because it's not passive, and it can focus on the weaker areas of the model too, so you can actually train on a larger dataset than the sum of recorded human knowledge.

IanCal 7mo ago

The best coding model won’t be the best roleplay one which won’t be the best at tool use. It depends what you want to do in order to pick the best model.

PhunkyPhil 7mo ago
I'm not saying you're wrong, but why is this the case? I'm out of the loop on training LLMs, but to me it's just pure data input. Are they choosing to include more code rather than, say fiction books?
- refulgentis 7mo ago
  I’ll go ahead and say they’re wrong (source: building and maintaining llm client with llama.cpp integrated & 40+ 3p models via http) I desperately want there to be differentiation. Reality has shown over and over again it doesn’t matter. Even if you do same query across X models and then some form of consensus, the improvements on benchmarks are marginal and UX is worse (more time, more expensive, final answer is muddied and bound by the quality of the best model)
  - SignalStackDev 7mo ago
    [dead]
  - IanCal 7mo ago
    Are you saying I’m wrong that some models are better for some tasks than others, but there isn’t a universally best model for all tasks?
  - stephenbez 7mo ago
    Thanks. Are there any links where I can learn more about this? I did some Googling and it appears that there are some examples where people say combining multiple models or multiple runs of the same models leads to improvements: https://www.sciencedirect.com/science/article/abs/pii/S0010482525010820 https://www.sciencedirect.com/science/article/abs/pii/S00104... https://arxiv.org/abs/2203.11171 https://arxiv.org/abs/2203.11171 But presumably people are less likely to publish a paper when an approach doesn’t work.
- jmalicki 7mo ago
  There is the pre-training, where you passively read stuff from the web. From there you go to RL training, where humans are grading model responses, or the AI is writing code to try to pass tests and learning how to get the tests to pass, etc. The RL phase is pretty important because it's not passive, and it can focus on the weaker areas of the model too, so you can actually train on a larger dataset than the sum of recorded human knowledge.