5 ms·
To me, mini releases matter much more and better reflect the real progress than SOTA models. The frontier models have become so good that it's getting almost i
by BoumTAC 6mo ago
To me, mini releases matter much more and better reflect the real progress than SOTA models.
The frontier models have become so good that it's getting almost impossible to notice meaningful differences between them.
Meanwhile, when a smaller / less powerful model releases a new version, the jump in quality is often massive, to the point where we can now use them 100% of the time in many cases.
And since they're also getting dramatically cheaper, it's becoming increasingly compelling to actually run these models in real-life applications.
- pzo 6mo agothey do are cheaper than SOTA but not getting dramatically cheaper but actually the opposite - GPT 5.4 mini is around ~3x more expensive than GPT 5.0 mini. Similarly gemini 3.1 flash lite got more expensive than gemini 2.5 flash lite.
- BoumTAC 6mo agoBut they are getting dramatically better. What's the point of a crazy cheap model if it's shit ? I code most of the time with haiku 4.5 because it's so good. It's cheaper for me than buying a 23€ subscription from Anthropic.
- philipkglass 6mo agoThe crazy cheap models may be adequate for a task, and low cost matters with volume. I need to label millions of images to determine if they're sexually suggestive (this includes but is not limited to nudity). The Gemini 2.0 Flash Lite model is inexpensive and performs well. Gemini 2.5 Flash Lite is also good, but not noticeably better, and it costs more. When 2.0 gets retired this June my costs are going up.
- dev_hugepages 6mo agoTime to gather a dataset and train your own model!
- brikym 6mo agoIf you're doing something common then maybe there are no differences with SOTA. But I've noticed a few. GPT 5.4 isn't as good at UI work in svelte. Gemini tends to go off and implement stuff even if I prompt it to discuss but it's pretty good at UI code. Claude tends to find out less about my code base than GPT and it abuses the any type in typescript.
- patates 6mo agoBig part of these differences may be the system prompts and/or the harness.
- sebastiennight 6mo ago> 100% of the time in many cases So, every single time, the new model works most of the time?
- kennywinker 6mo agoYou’ve parsed the sentence wrong. Read it as: “You can use them full time in many cases”
- zozbot234 6mo ago> And since they're also getting dramatically cheaper, it's becoming increasingly compelling to actually run these models in real-life applications. They're not really cheaper than the SOTA open models on third-party inference platforms, and they're generally dumber. I suppose they're still worth it if you must minimize latency for any given level of smarts, but not really otherwise.
- XCSme 6mo agoWell, in that case, the difference is quite minimal between 5 mini and 5.4 mini 5.4 mini seems to be a lot more wild/unstable, but with this instability it gets the right answer more often. https://aibenchy.com/compare/openai-gpt-5-4-mini-medium/openai-gpt-5-mini-medium/ https://aibenchy.com/compare/openai-gpt-5-4-mini-medium/open...
- ainch 6mo agoI use Gemini via its web app, which aggressively autoswitches to the Flash over Pro, but I usually notice quickly because the answers are weird or the logic doesn't quite follow. I feel like, at least for 'daily driver' usage, small models are still a little disappointing. That said, they're getting very good for more automation-y tasks with simple, well-constrained tasks.
- mountainriver 6mo agoMost annoying part of their web app and a really terrible idea. I often just think Gemini is terrible but then it turns out they silently changed the model on me
- ainch 6mo agoI got Codex to whip me up a Chrome extension that autoswaps back to Pro whenever I reload the page. It's made Gemini significantly less irritating to use.