5 ms·

Wait WTF?! I thought performance was, well, performance, not efficacy! OpenRouter's UI in this section is remarkably broken and unclear at the same time, and I

by bbor 5d ago

Wait WTF?! I thought performance was, well, performance, not efficacy! OpenRouter's UI in this section is remarkably broken and unclear at the same time, and I have no idea where the author got those clean charts. "AutoExacto" numbers (completely meaningless name) are available for 6 providers, two of which are the same provider, and none of which are even in the top half throughput-wise. When you click "+28 more providers", it just shows a clearly broken modal. And the the next two graphs are even worse. Really all of these graphs are pretty and completely 100% useless. I feel like it's absolutely insane that some providers serve the same model with much less efficacy. That doesn't make sense to me. What's going on?! I'm suddenly feeling intense shame for having routed all my non-subscription usage through them so far, and honestly some white hot anger that they would blatantly lie about something so important. What am I missing? Is this really true?

numlocked 5d ago
We run benchmarks against all of our endpoints, in production. That first chart that the author shows is in fact our live benchmarking data. If providers underperform, we kick them out of the routing pool. That is why we run those benchmarks. Performance And errrr...yeah...that auto-exacto performance chart is both 100% useless, and totally unclear. We will get that fixed. But under the covers it is doing a lot of valuable work! https://openrouter.ai/docs/guides/routing/auto-exacto https://openrouter.ai/docs/guides/routing/auto-exacto
- bbor 4d ago
  Oh wow someone from the team! I'm really heartened by this response -- as you can tell, I've been something of a fan, so I felt kinda betrayed by the idea that this was a risk. Thanks for clarifying, and for taking my frustrated hyperbole in stride. Submitting an application to your product role now. "All of the models are at least okay" is heartening, but there's a whole bunch of fun places to take this. ¡Viva La OpenRouter! (again)

by bbor 5d ago

numlocked 5d ago

We run benchmarks against all of our endpoints, in production. That first chart that the author shows is in fact our live benchmarking data. If providers underperform, we kick them out of the routing pool. That is why we run those benchmarks. Performance And errrr...yeah...that auto-exacto performance chart is both 100% useless, and totally unclear. We will get that fixed. But under the covers it is doing a lot of valuable work! https://openrouter.ai/docs/guides/routing/auto-exacto https://openrouter.ai/docs/guides/routing/auto-exacto

bbor 4d ago
Oh wow someone from the team! I'm really heartened by this response -- as you can tell, I've been something of a fan, so I felt kinda betrayed by the idea that this was a risk. Thanks for clarifying, and for taking my frustrated hyperbole in stride. Submitting an application to your product role now. "All of the models are at least okay" is heartening, but there's a whole bunch of fun places to take this. ¡Viva La OpenRouter! (again)