4 ms·
Yes to all this but more. The thing that made me leave and go to a single provider was token caching. I have to keep blocking providers that don't properly cach
by jmward01 5d ago
Yes to all this but more. The thing that made me leave and go to a single provider was token caching. I have to keep blocking providers that don't properly cache. I see performance tank and then I look in the logs and a new provider has been rotated in and every call to them is uncached because they are clearly broken. This has happened a few times now and essentially destroys cost savings (these providers also often have terrible quality). Don't they monitor for simple things like this? Their own logs show how clear this pattern is for some providers. Simple cache % stats would allow them to block providers nearly instantly.
- numlocked 5d agoWe have made MASSIVE improvements here, and network-wide caching rates have been improving relentlessly. We do publish the cache rates for each endpoint; see the "performance" area of our model pages. E.g. https://openrouter.ai/deepseek/deepseek-v4-flash-0731#performance https://openrouter.ai/deepseek/deepseek-v4-flash-0731#perfor... Open to feedback on how to make this better.
- stavros 5d agoCan I block endpoints whose cache rates are below a certain percentage?
- ElectricalUnion 5d agoCan I block providers (for model, not in general) that have sightly cheaper input/output tokens but have more that 10x the average cache cost? Can I block providers (for a model, not in general) that set up cache write cost when the mode is free cache writes?
- randomblock1 5d agoYou know how there's a router mode to use the cheapest provider? That only takes into account uncached rates, last I checked. Make another one that takes into account effective rates (the ones that include cache).
- jmward01 5d agoI think the future -may- be a hosted routing client. Something similar to an auth client app. Why? A few reasons: - Customers being able to decide their own routing with true logic is a huge feature. Open router provides the seamless switching/api, route switching decisions are available in a client. - Similarly, providing hooks at this level gives a chance for stats/other things that are hard to plug into prod code elsewhere - a true middle man hosting for other things like MCP may also turn into a real win once it is implemented. Just a random thought though. My point about quality/cost being clobbered by bad providers remains. The fact that cache and quality is badly handled makes me doubt that training data choices are being respected. You need a more public trust/certification process for providers with real teeth when they cheat. I'm going to wait a bit to see how things evolve and check back later.
- kroaton 4d agoYou need to validate what providers are actually serving. Add benchmarks, properly showcase what quantization they are serving on the model and KV cache, etc. Until that happens, your service is doomed to be shitty.