7 ms·

This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapsho

This squares with my, much much, smaller OpenRouter usage. It’s just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found. OpenRouter sells the idea of swapping being commodity providers but it couldn’t be further from the truth. Provider A is often not swappable for B or C (again, as this author found). It can be crazy-making as you sit there thinking “OpenRouter has no clothes right?! Am I the one that’s wrong?”. I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly. Without pinning you are in for a world of hurt and unreliability (varying model capabilities, speed, etc).

Mairoce 5d ago
Yep, I wish more people realized that OpenRouter’s sole value proposition is that they are an LLM wholesaler.
fc417fc802 5d ago
> to the point I wonder if I should just use the providers directly. How many account credentials, balances, and tokens do you want to maintain? Even without automatic failover services such as openrouter are still incredibly useful. Personally I pin a single vetted provider in the interest of minimizing risk.
embedding-shape 5d ago
> just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found. But that's the intention right? Even the name implies they just send stuff around for you, and if you want to control the routing, you'd lock down providers. I don't see how they could build what they wanted to build, and not have it end up unreliable if you freely round-robin between providers, it's bound to work exactly like this. > I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly. This is quite literally the point of OpenRouter. A unified interface, so you can easily switch providers without changing a ton of code which using providers directly would most likely mean, as there are slight differences between them. And the providers all run different weights, so of course quality/performance will differ among them. I guess OpenRouter is a bit like Amazon, in that they're just routing stuff around for you, but to actually find the good and usable stuff, you need to focus in on what providers/manufacturers you know are good, and stick with those. Still, the unified interface helps you to shop around and try different ones when you want to.
- bbor 5d ago
  if you want to control the routing, you'd lock down providers I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate. If OpenRouter is a marketplace to pick a provider while avoiding huge problems, it is terrible at that job. It surfaces literally none of that info in the top-level list, the graphs below are mislabeled and useless at best, and doesn't notify you of this horrifying situation anywhere, even in passing. There's not even a way to compare providers, AFAICT -- you can only compare models. This is quite literally the point of OpenRouter. Their tagline is "better prices, better uptime, no subscriptions". The first two of these directly and inherently contradict your understanding -- neither would be possible if OpenRouter was just a fancy way to change something in their GUI rather than changing the target url of your gateway.
  - embedding-shape 5d ago
    > I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate. Why do you care about the public benchmarks at all? The way companies and effective individual developers use OpenRouter, is that you first create your evaluation framework/benchmark, for your specific tasks and use cases, and make that real easy to run and use various models and providers with it. Then you run this to gather data. Then you use said data to figure out what works and what the quality/cost tradeoff you want to make is. Then you lock that down in production while you keep iterating on your benchmark to make it match with real-world use cases and keep adding the new models that pop up. I don't think anyone serious is just willy-nilly making individual requests against OpenRouter and similar platforms, get a "feel for a provider" then use only that provider. Not only would it be wildly inefficient, but also you need hard numbers to compare so you can make informed choices. For this process and workflow, OpenRouter is great, because adding/changing providers and models is essentially changing two strings, rather than having a adapter for each platform you want to try out. If you just want best accuracy requests from SOTA models for your agent you run locally or whatever, then don't use OpenRouter, it doesn't make much sense, but use the provider the model maker has available, as almost all of them run their own endpoints.
miki123211 5d ago
I think Openrouter is great for quick testing, getting a "feel" for the model, or very quick integration jobs where you want to try out as many models as possible. For truly production use cases, use Novita, Fireworks, Toghether or something of the sort.
- nacs 5d ago
  Did you miss the part of this article where the benchmarks specifically call out Fireworks as one of the worst in their tests: > Fireworks scored 46% on TAU, a 30 point gap Another surprise was DigitalOcean being bottom of barrel too. Companies are apparently willing to risk their brand name by being deceptive about these heavily quantized/flawed model-serving.
  - svachalek 5d ago
    DigitalOcean (at least through OpenRouter) is pretty reliably bad in my experience. Fireworks can be great, but it depends on the model and on the day.
    - tandr 5d ago
      DigitalOcean is bad not just through OR... Direct accesss through DO gave me such a miniscule context and maximum tokens, it was pretty much unusable. So, accessing it through OpenRouter gave bigger context length, but model behaves like it was seriously damaged - tool calling was producing paths that had / replaced with some other symbols, files were not found etc., and this is just for things that errored out, I have no idea how bad reasoning was. I had to blacklist DO completely on OpenRouter.
  - FergusArgyll 5d ago
    It's not always deceptive. Bugs in inference can be very subtle (well, bugs everywhere can be subtle). The providers should be benchmarking their offerings daily
artursapek 5d ago
I only use OpenRouter to benchmark models. For my production app, I have direct integrations with OpenAI, Anthropic, Google, xAI. Even just input caching is enough of a reason to do that, assuming you are trying to build the fastest and most cost-efficient product possible. I don't understand how people supposedly run production apps using OpenRouter unless those two variables somehow don't matter to them.
- drakmail 5d ago
  Not every usecase requires prompt caching, for example prompt enchantment jobs usually can't benefit from caching at all
kadoban 5d ago
You know you can define your own provider filters and orderings, right? The filters and such are not _that_ advanced, but it might do what you need if you haven't already tried that.
Aurornis 5d ago
I treat OpenRouter as a central point to access specific providers. Pinning is a given for that use case. I’ve tried using un-pinned models and the experience is exactly as you described: Some providers are so unreliable that the majority of requests fail. Some providers do weird things like abruptly end the response (which I get billed for and have to re-submit). Some providers are clearly running heavily quantized versions of the model because their eval performance is terrible. Some providers advertise features on OpenRouter but will reject those requests when submitted to their API. So pinning is the way to go.
- highfrequency 5d ago
  What providers do you find most reliable? And is that general across models?
random3 5d ago
This (by definition) the reality of every normalization/proxy layer, standard, etc. There’s always a cost for flexibility/freedom. What’s traded is when and by whom it’s paid.
arjie 5d ago
OpenRouter has the advantage that I can maintain a single balance across model providers so I use it for quick iteration across and then write it into my own router (required anyway since I have many on-prem models running). The portable balance is actually very useful and my credit card is in one place. If I could pay per request without maintaining a balance or credit card out of a single wallet (using crypto or something maybe) I would happily simply write the integration myself because OR’s caching is often not as good without some hoop jumping.
szundi 5d ago
[dead]

Mairoce 5d ago

Yep, I wish more people realized that OpenRouter’s sole value proposition is that they are an LLM wholesaler.

fc417fc802 5d ago

> to the point I wonder if I should just use the providers directly. How many account credentials, balances, and tokens do you want to maintain? Even without automatic failover services such as openrouter are still incredibly useful. Personally I pin a single vetted provider in the interest of minimizing risk.

embedding-shape 5d ago

> just incredibly unreliable and you are forced to pin providers and even then it can be a crapshoot as the author found. But that's the intention right? Even the name implies they just send stuff around for you, and if you want to control the routing, you'd lock down providers. I don't see how they could build what they wanted to build, and not have it end up unreliable if you freely round-robin between providers, it's bound to work exactly like this. > I love the _idea_ of OpenRouter and maybe Stripe can improve this situation but the only sane way I’ve found to use it is to tightly pin providers to the point I wonder if I should just use the providers directly. This is quite literally the point of OpenRouter. A unified interface, so you can easily switch providers without changing a ton of code which using providers directly would most likely mean, as there are slight differences between them. And the providers all run different weights, so of course quality/performance will differ among them. I guess OpenRouter is a bit like Amazon, in that they're just routing stuff around for you, but to actually find the good and usable stuff, you need to focus in on what providers/manufacturers you know are good, and stick with those. Still, the unified interface helps you to shop around and try different ones when you want to.

bbor 5d ago
if you want to control the routing, you'd lock down providers I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate. If OpenRouter is a marketplace to pick a provider while avoiding huge problems, it is terrible at that job. It surfaces literally none of that info in the top-level list, the graphs below are mislabeled and useless at best, and doesn't notify you of this horrifying situation anywhere, even in passing. There's not even a way to compare providers, AFAICT -- you can only compare models. This is quite literally the point of OpenRouter. Their tagline is "better prices, better uptime, no subscriptions". The first two of these directly and inherently contradict your understanding -- neither would be possible if OpenRouter was just a fancy way to change something in their GUI rather than changing the target url of your gateway.
- embedding-shape 5d ago
  > I don't want to control routing. I want the model to work how the giant, prominent "BENCHMARKS" section says it works, not randomly have a 100x error rate. Why do you care about the public benchmarks at all? The way companies and effective individual developers use OpenRouter, is that you first create your evaluation framework/benchmark, for your specific tasks and use cases, and make that real easy to run and use various models and providers with it. Then you run this to gather data. Then you use said data to figure out what works and what the quality/cost tradeoff you want to make is. Then you lock that down in production while you keep iterating on your benchmark to make it match with real-world use cases and keep adding the new models that pop up. I don't think anyone serious is just willy-nilly making individual requests against OpenRouter and similar platforms, get a "feel for a provider" then use only that provider. Not only would it be wildly inefficient, but also you need hard numbers to compare so you can make informed choices. For this process and workflow, OpenRouter is great, because adding/changing providers and models is essentially changing two strings, rather than having a adapter for each platform you want to try out. If you just want best accuracy requests from SOTA models for your agent you run locally or whatever, then don't use OpenRouter, it doesn't make much sense, but use the provider the model maker has available, as almost all of them run their own endpoints.

miki123211 5d ago

I think Openrouter is great for quick testing, getting a "feel" for the model, or very quick integration jobs where you want to try out as many models as possible. For truly production use cases, use Novita, Fireworks, Toghether or something of the sort.

nacs 5d ago
Did you miss the part of this article where the benchmarks specifically call out Fireworks as one of the worst in their tests: > Fireworks scored 46% on TAU, a 30 point gap Another surprise was DigitalOcean being bottom of barrel too. Companies are apparently willing to risk their brand name by being deceptive about these heavily quantized/flawed model-serving.
- svachalek 5d ago
  DigitalOcean (at least through OpenRouter) is pretty reliably bad in my experience. Fireworks can be great, but it depends on the model and on the day.
  - tandr 5d ago
    DigitalOcean is bad not just through OR... Direct accesss through DO gave me such a miniscule context and maximum tokens, it was pretty much unusable. So, accessing it through OpenRouter gave bigger context length, but model behaves like it was seriously damaged - tool calling was producing paths that had / replaced with some other symbols, files were not found etc., and this is just for things that errored out, I have no idea how bad reasoning was. I had to blacklist DO completely on OpenRouter.
- FergusArgyll 5d ago
  It's not always deceptive. Bugs in inference can be very subtle (well, bugs everywhere can be subtle). The providers should be benchmarking their offerings daily

artursapek 5d ago

I only use OpenRouter to benchmark models. For my production app, I have direct integrations with OpenAI, Anthropic, Google, xAI. Even just input caching is enough of a reason to do that, assuming you are trying to build the fastest and most cost-efficient product possible. I don't understand how people supposedly run production apps using OpenRouter unless those two variables somehow don't matter to them.

drakmail 5d ago
Not every usecase requires prompt caching, for example prompt enchantment jobs usually can't benefit from caching at all

kadoban 5d ago

You know you can define your own provider filters and orderings, right? The filters and such are not _that_ advanced, but it might do what you need if you haven't already tried that.

Aurornis 5d ago

I treat OpenRouter as a central point to access specific providers. Pinning is a given for that use case. I’ve tried using un-pinned models and the experience is exactly as you described: Some providers are so unreliable that the majority of requests fail. Some providers do weird things like abruptly end the response (which I get billed for and have to re-submit). Some providers are clearly running heavily quantized versions of the model because their eval performance is terrible. Some providers advertise features on OpenRouter but will reject those requests when submitted to their API. So pinning is the way to go.

highfrequency 5d ago
What providers do you find most reliable? And is that general across models?

random3 5d ago

This (by definition) the reality of every normalization/proxy layer, standard, etc. There’s always a cost for flexibility/freedom. What’s traded is when and by whom it’s paid.

arjie 5d ago

OpenRouter has the advantage that I can maintain a single balance across model providers so I use it for quick iteration across and then write it into my own router (required anyway since I have many on-prem models running). The portable balance is actually very useful and my credit card is in one place. If I could pay per request without maintaining a balance or credit card out of a single wallet (using crypto or something maybe) I would happily simply write the integration myself because OR’s caching is often not as good without some hoop jumping.

szundi 5d ago

[dead]