5 ms·
It's scary relying on Google's models. I have a very price sensitive workload that used to run on flash 2.5 lite - it's deprecated now. The replacement 3.1 fl
by swe_dima 2mo ago
It's scary relying on Google's models.
I have a very price sensitive workload that used to run on flash 2.5 lite - it's deprecated now.
The replacement 3.1 flash lite is a lot more expensive, but now also has a sunset date.
3.5 flash lite is even more expensive.
So the price is rising and you have no choice but to keep paying more and more.
- zuzululu 2mo agosame I just switched to OpenAI after using flash 2.5 lite for almost everything at our company. We spent thousands just to build this workflow now Google says screw off
- rayboy1995 2mo agoI moved directly from 2.5 flash lite to deepseek v4 flash, its already cheaper and if your prompt caching is good you can save so much more money.
- binary132 2mo agocould you explain how to optimize prompt caching or point to a doc about it?
- NeutralForest 2mo agoAnything Sam Rose is worth reading: https://ngrok.com/blog/prompt-caching https://ngrok.com/blog/prompt-caching but the implementation will be up to your provider and harness, for deepseek, they expose some numbers: https://api-docs.deepseek.com/guides/kv_cache/ https://api-docs.deepseek.com/guides/kv_cache/ and Anthropic has a list of actions invalidating your cache: https://platform.claude.com/docs/en/build-with-claude/prompt-caching#what-invalidates-the-cache https://platform.claude.com/docs/en/build-with-claude/prompt... Basically, you avoid anything dynamic: model change, tool change, etc it's also important that your system prompt or main prompt doesn't have non-static data like the date/time/place or someone's name (the person you interact with in a chatbot for example). That should be left to tool call or search.
- arjie 2mo agoI just put the varying parameters in a trailer prompt and have them change every time. It doesn’t matter because the cache is prefix keyed. You lose caching for the last 20 tokens or so but that’s not a big deal. Moving it to a tool call makes it too slow (needs full roundtrip). If you’re constructing the prompt you don’t have to jam everything together you can arrange it appropriately.
- NeutralForest 2mo agoYes indeed! Mostly don't put changing data in the beginning or prepend.
- samwho 2mo agoSam Rose here. Thank you <3
- insane_dreamer 2mo agosamwho? samrose.
- NeutralForest 2mo agoThe man himself, thank you for the articles =)
- samwho 2mo agoYou are extremely welcome.
- deleted 2mo ago[deleted]
- apwheele 2mo agoNot an open source, but I discuss it in my book with examples for OpenAI/Anthropic/Gemini, https://crimede-coder.com/blogposts/2026/LLMsForMortals https://crimede-coder.com/blogposts/2026/LLMsForMortals. All of the models, you need to have a consistent input to get the cache hit. So if you are chatting with a document, and change the system prompt, it will be a cache miss, even if the rest of the items are all the same. If you even pass in the document in not the same order as the prompts, it will be a cache miss. Or if you add tool calls or structured outputs, it will be a cache miss. (Since those generally go at the beginning of the prompt call, not at the end.) Most of the time when reading documents from URLs directly it will never cache. (Need to typically pass in the bytes directly, or use the provider document store index.) Gemini has a 4096 minimum token size with the 3 version models before even getting a cache hit. OpenAI it is lower (1024), and is automatic, but only happens in increments of 124. Anthropic can also get cache hits at 1024 tokens, but you need to explicit ask for it (and pay extra). Caching by default typically lives for 5 minutes since the last cache hit across providers. But some of them you can ask for longer. AWS for Anthropic models can be tricky with multiple endpoint routing, so can get cache misses if it happens to route to a different endpoint.
- hagen8 2mo agoJust switch the model, its not that much effort tbh. And u can also get a cheaper model than 2.5 lite for the same intelligence
- tacoooooooo 2mo agoits not always that simple. dropping in a new model is trivial, but highly specific workflows may rely on specific _invisible_ aspects of a model. when that model gets deprecated, the workflow needs to be rebuilt/re-tuned to work with a different model. google's inability or unwillingness to provide stable timelines for model deprecation makes it risky to build complex workflows using their models
- written-beyond 2mo ago100% agreed in the same boat right now. Feeling really screwed over by Google rn
- aitchnyu 2mo agoLoad-bearing (whoops) quirks were noticeable months back, but haven't most flagship models become predictable and reliable?
- tacoooooooo 2mo agoit does seem to be moving in that direction. There were really specific things (large, complex json outputs) that gemini-2.5 flash was basically the only model that seemed capable of reliably for a long period. gpt-5+ has covered the usecase for us now pretty well but still evals slightly below what 2.5 could do
- ActivePattern 2mo agoYou would be surprised how much of a difference the model makes for certain niche tasks. For my use case, `gemini-3.1-flash-lite` is ~20% higher accuracy than the next best model of comparable cost (considering both proprietary and open-weight alternatives)
- ElFitz 2mo agoThat’s part of why, since Firebase, I’ve tried to never depend on Google products for business, especially not GCP. Features stay in Beta for ages, whatever that actually means, and released ones get deprecated things fast. Where some of the competitions treats deprecating entire services as "let’s not put it on your frontpage, put deprecation notices all over the doc, and politely ask new users not to start new project with them".
- superkuh 2mo agoI felt the same way about openai's text-davinci-002 and code-davinci-002 (gpt-3.5). They were amazing completion models and openai basically dumped them with no equal cost or equal performance replacement. Instead all their models are opaque with no ability to work in completion mode where one actually controls the text input to the model. These days no company even has completion models where one controls the text input fully. Worthless.
- raducu 2mo ago> So the price is rising and you have no choice but to keep paying more and more. I presume you can't use deepseek?
- stiltzkin 2mo ago[dead]
- Cyclone_ 2mo agoThey know that there's big enterprises that will have a strong preference to work with another big enterprise instead of relying on a younger company. At least that's why I think they believe they can do this sort of thing and get away with it.
- viccis 2mo ago>So the price is rising and you have no choice but to keep paying more and more. You can also just write code like you did a year or two ago.
- anthonypasq 2mo ago[flagged]
- Mistletoe 2mo agoCan you give examples of other things they can do that would be worth paying for?
- viccis 2mo agoMostly because the person I was replying to has commented about using it to write code. If you're using it for other purposes, then I give you permission to ignore my comment; there's no reason to descend into name calling.
- anthonypasq 2mo ago
- 5701652400 2mo agosame here. our production workloads was on Gemini for 2 years. seeing Google unilaterally dropping perfectly fine models and charing you 50x more for worse results is not good. we are switching to Deepseek.
- h2aichat 2mo agoOpencode Go is just the same. Each month I will I can do less. Dont ask me why?
- pdntspa 2mo agoI'm running price-sensitive data extraction workloads on flash 2.5 and its still the king when it comes to accuracy + cost, all the gemini 3 variants perform a bit worse and cost a lot more. Low-key freaking out, ngl
- purvahk08 2mo agoyou should try out scaledown, it's 29x cheaper and 9% more accurate than gemini https://scaledown.ai/benchmarks/scaledown-vs-gemini https://scaledown.ai/benchmarks/scaledown-vs-gemini
- thinkingtoilet 2mo agoAll models are increasing in price. Everything up to now has been subsidized by investors, private and public.
- greatgib 2mo agoAnd somehow, the most annoying is not even the price hike, but it is that is you expect to build a product on any of theirs models, they spend their time being deprecated and you have like to be on the lookup to start from scratch selecting a model and fitting it every year or so... Impossible to have any stability...