5 ms·

I love OpenRouter, long time user. Stripe will hopefully be a good custodian. I just want to point out some features of OpenRouter that make it more than just

by nikcub 28d ago

I love OpenRouter, long time user. Stripe will hopefully be a good custodian. I just want to point out some features of OpenRouter that make it more than just a model selection and routing endpoint and that I find incredibly useful: 0/ Default routing is to the cheapest provider, but they're usually not the most performant. I'd guess 99% of OpenRouter integrations never tweak the default routing. Here you can setup cheapest with performance minimums: https://openrouter.ai/docs/guides/routing/provider-selection#use-case-2-find-the-cheapest-model-meeting-performance-requirements https://openrouter.ai/docs/guides/routing/provider-selection... You can also stack model selection in priority 1/ Using broadcast you can push all your analytics to clickhouse / s3 / snowflake and a bunch of other compatible destinations. Setup a clickhouse server ($5 VPS[0]) and send all your traces to it: https://openrouter.ai/docs/guides/features/broadcast https://openrouter.ai/docs/guides/features/broadcast customise your own observability in your dashboards from there. Superwin 2/ Model router is also a natural home for llm security - OpenRouter has the beginnings of prompt injection detection: https://openrouter.ai/docs/guides/features/guardrails/prompt-injection https://openrouter.ai/docs/guides/features/guardrails/prompt... there is also PII detection. This will show up in observability as rejections/blocks etc. There are so many model routing solutions (same with observability, security etc.) but they're all 80% solutions - OpenRouter really rounds out with well implemented features that you need when deploying models at any scale and I gladly pay the toll. [0] not sure if these exist any more but clickhouse is resource efficient

deleted 28d ago
[deleted]
sbinnee 28d ago
I have been using openrouter only superficially. I didn’t know about the broacasting feature. Thanks for that.
dayeye2006 28d ago
noob q -- does the routing hurt cache hit rate?
- miroljub 28d ago
  Of course, of routing the same session to different providers. But don't worry, many providers there charge for cache the same price like non cached ;)
- Ey7NFZ3P0nzAe 28d ago
  By default yes bit you can pin providers to avoid this.
pbreit 28d ago
This seems like something xAi could reproduce in a weekend
- rswail 28d ago
  Could but hasn't. What exactly is xAI's USP compared to the others?
  - cootsnuck 28d ago
    CSAM and having the potential to randomly becoming unhinged due to strong alignment with the whims of the one dude? Surely puts enterprises at ease.
  - pbreit 27d ago
    Copying others better and faster. I would say this is right up their alley.
- michaelt 28d ago
  The other thing OpenRouter gets you is: Access to loads of models via a common API/sign-up/prepayment mechanism. I want to know which vendor/model does best at my extracting-facts-from-text task? Which does best at my OCR-a-text-document task? Which can deal with a safe-for-work beach photo without a censorship system false alarm? OpenRouter lets me run my tests against openai and anthropic and google and x and bytedance and qwen and llama, with a single sign-up and a single payment.
  - catlifeonmars 28d ago
    You can do that with AWS too.
    - walthamstow 28d ago
      AWS Bedrock's model offering is tiny compared to Open Router. I have to agree with GP, there's no better way to run an eval suite against many models
troupo 28d ago
> features of OpenRouter that make it more than just a model selection and routing endpoint All those features show it's just a model selector and router.
- johnbarron 28d ago
  Meaning you can switch in seconds. Stripe just has too much money...
hobofan 28d ago
> Model router is also a natural home for llm security - OpenRouter has the beginnings of prompt injection detection Maybe I'm biased from the perspective of a "harness provider", but I think Model routers often have too little context to act as well-informed prompt injection prevention. e.g. it lacks context of where which part of the message(s) originates from and sanitization/safeguards were already performed on the application layer. Something like OpenRouter's "flag" mode is fine, but usage of auto-redact or auto-block should really only be used if there is significant risk exposure through your harness or otherwise they are a constant source of bugs.
johnbarron 28d ago
They might be popular for indie developers, but nobody doing serious AI or in a corporate environment is using them. And if they are, their compliance team is about to strike them down. The VCs forcing this acquisition do know this. - Why would you add a penalty of 50 ms at a minimum? And that is not the p95... Just run LiteLLM in house and you dont even really need that. - Their capacity pools are shared across the whole user base, a massive batch processing by another of their customers and think what that means for your response time... - So instead of negotiating corporate rates with OpenAI or Anthropic, you would be using an intermediary and topping up the corporate credit card... for a 5% markdown ? Really? - They can see all your critical corporate data on the in and out - They present some pink SOC 2 promises but then wash their hands and defer to you and the providers. Its just the Bolt and Uber model the drivers are not our employees.... - They are a man in the middle proxy that is a massive security liability for your corporation - They have no support for private cloud points - No geofencing guarantees - No intellectual property legal indemnification unlike what AWS or Microsoft or Google offers - Its a provider roulette inconsistent with hosts providing different quantization levels causing random shifts in response quality - Support via a Discord server... The only reason they were not shutdown yet by Anthropic or OpenAI is because they have the same VCs, as those two. That would mean said VCs investment would go to zero. Oh and those are the same VCs that own Stripe... Just setup a private proxy tier using something like LiteLLM, even if you really dont need it. Just code your enterprise apps to have have fallback loops on the core hyperscaler providers like AWS Bedrock or Azure Foundry...
- devy 28d ago
  Yep. https://news.ycombinator.com/item?id=49344067 https://news.ycombinator.com/item?id=49344067
- MallocVoidstar 28d ago
  > Why would you add a penalty of 50 ms at a minimum? All of the customer service bots I've dealt with recently were reasoning models that thought for multiple seconds. I would not have noticed 50ms extra.
- burnto 28d ago
CurbStomper 28d ago
[dead]

by nikcub 28d ago

deleted 28d ago

[deleted]

sbinnee 28d ago

I have been using openrouter only superficially. I didn’t know about the broacasting feature. Thanks for that.

dayeye2006 28d ago

noob q -- does the routing hurt cache hit rate?

miroljub 28d ago
Of course, of routing the same session to different providers. But don't worry, many providers there charge for cache the same price like non cached ;)
Ey7NFZ3P0nzAe 28d ago
By default yes bit you can pin providers to avoid this.

pbreit 28d ago

This seems like something xAi could reproduce in a weekend

rswail 28d ago
Could but hasn't. What exactly is xAI's USP compared to the others?
- cootsnuck 28d ago
  CSAM and having the potential to randomly becoming unhinged due to strong alignment with the whims of the one dude? Surely puts enterprises at ease.
- pbreit 27d ago
  Copying others better and faster. I would say this is right up their alley.
michaelt 28d ago
The other thing OpenRouter gets you is: Access to loads of models via a common API/sign-up/prepayment mechanism. I want to know which vendor/model does best at my extracting-facts-from-text task? Which does best at my OCR-a-text-document task? Which can deal with a safe-for-work beach photo without a censorship system false alarm? OpenRouter lets me run my tests against openai and anthropic and google and x and bytedance and qwen and llama, with a single sign-up and a single payment.
- catlifeonmars 28d ago
  You can do that with AWS too.
  - walthamstow 28d ago
    AWS Bedrock's model offering is tiny compared to Open Router. I have to agree with GP, there's no better way to run an eval suite against many models

troupo 28d ago

> features of OpenRouter that make it more than just a model selection and routing endpoint All those features show it's just a model selector and router.

johnbarron 28d ago
Meaning you can switch in seconds. Stripe just has too much money...

hobofan 28d ago

> Model router is also a natural home for llm security - OpenRouter has the beginnings of prompt injection detection Maybe I'm biased from the perspective of a "harness provider", but I think Model routers often have too little context to act as well-informed prompt injection prevention. e.g. it lacks context of where which part of the message(s) originates from and sanitization/safeguards were already performed on the application layer. Something like OpenRouter's "flag" mode is fine, but usage of auto-redact or auto-block should really only be used if there is significant risk exposure through your harness or otherwise they are a constant source of bugs.

johnbarron 28d ago

They might be popular for indie developers, but nobody doing serious AI or in a corporate environment is using them. And if they are, their compliance team is about to strike them down. The VCs forcing this acquisition do know this. - Why would you add a penalty of 50 ms at a minimum? And that is not the p95... Just run LiteLLM in house and you dont even really need that. - Their capacity pools are shared across the whole user base, a massive batch processing by another of their customers and think what that means for your response time... - So instead of negotiating corporate rates with OpenAI or Anthropic, you would be using an intermediary and topping up the corporate credit card... for a 5% markdown ? Really? - They can see all your critical corporate data on the in and out - They present some pink SOC 2 promises but then wash their hands and defer to you and the providers. Its just the Bolt and Uber model the drivers are not our employees.... - They are a man in the middle proxy that is a massive security liability for your corporation - They have no support for private cloud points - No geofencing guarantees - No intellectual property legal indemnification unlike what AWS or Microsoft or Google offers - Its a provider roulette inconsistent with hosts providing different quantization levels causing random shifts in response quality - Support via a Discord server... The only reason they were not shutdown yet by Anthropic or OpenAI is because they have the same VCs, as those two. That would mean said VCs investment would go to zero. Oh and those are the same VCs that own Stripe... Just setup a private proxy tier using something like LiteLLM, even if you really dont need it. Just code your enterprise apps to have have fallback loops on the core hyperscaler providers like AWS Bedrock or Azure Foundry...

devy 28d ago
Yep. https://news.ycombinator.com/item?id=49344067 https://news.ycombinator.com/item?id=49344067
MallocVoidstar 28d ago
> Why would you add a penalty of 50 ms at a minimum? All of the customer service bots I've dealt with recently were reasoning models that thought for multiple seconds. I would not have noticed 50ms extra.
burnto 28d ago

CurbStomper 28d ago

[dead]