7 ms·
Not sure about the level of irony here, but I keep hearing models have plateaued since a while now, but I keep being impressed with the latest model performance
by lukan 4d ago
Not sure about the level of irony here, but I keep hearing models have plateaued since a while now, but I keep being impressed with the latest model performance.
- unleashhale 4d agoAnything in particular? My experience has been like seeing the addition of retractable cupholders, but maybe different domains.
- qskousen 4d agoI have a pet project I have been working away on for some time that involves building GPU backends for various cards in Zig, lots of complex stuff in it. Lately I mostly use Opus 5, it can pretty reliably plug away at things but it does mess stuff up occasionally. For this codebase, Fable 5.1 was noticeably better at getting things right and doing things in a good reliable way. Of course, I can only use Fable for a bit before I hit the usage cap for the week, so I save it for the tougher things. That said, I absolutely abhor the way recent Anthropic models write prose, especially comments. I recently tried doing a fairly normal task for this codebase with codex, as I have seen a lot of people talking it up on here. A single task running for ~1-2 hours burned through over half of my usage for the week on the $125/month plan, not on a top model (I don't remember which one specifically I used). It struggled to get the basics done, then got absolutely stuck on a follow up. Handed it over to Claude and it 1-shot it.
- eru 4d agoI really liked codex in the last few weeks, especially its ability to clean up after Claude's (prose) messes and do reviews. But in the last few days something seems to have happened that made Codex's models massively stupider (for what I am doing). Really weirdly, it suddenly refused to even run tests it previously wrote itself (and previously ran), because of some false positive about cybersecurity. That by itself is not evidence of stupidity. Trying to make a 200+ file PR full of research notes is, and the PR didn't even solve the problem I asked it to.
- ipaddr 4d agoThat's the case for open weights models. Hosted Deepseek Flash 731 copy isn't changing randomly one day because the parent company decided to change it.
- drTobiasFunke 4d agoReally? My employer rolled back to opus 4.8 because 5 was expensive AND crap. Didnt even consider fable because it didn’t add any additional value. For most software eng and design work opus 4.6-4.8 just works fine. For everyday joe asking ai to plan a trip or home diy work even sonnet works fine. Any cybersecurity or other areas are niches that cannot support trillion $ valuations. What am I missing? Genuinely curious
- bpodgursky 4d agoIf Fable doesn't add additional value in your workplace, it means you aren't being ambitious enough in how you integrate agents into your workstream. Yes, it's probably comparable to 4.8 if you are just using it to write code and put up a couple pull requests. That's not where things are now.
- bix6 4d agoAnd where are things now?
- nradov 4d agoYou shouldn't be down voted, AI native companies have already moved up to the next level beyond writing individual PRs.
- Tanjreeve 4d ago"AI Native" here meaning Companies where your token use isn't scrutinised/capped yet?
- nozzlegear 4d agoThis is just "you're holding it wrong" with a little smooch of condescension. If only we plebeians could comprehend what magnificent works those who have ambitiously integrated agents into the workstream have wrought!
- bpodgursky 4d ago
- 0xcde4c3db 4d agoI don't think "plateaued" is the right word, but I do feel like there's been something like a logistic curve compression in the difference between smaller and larger models as the field evolves. For inference at least, the scale of practical difference between a single high-VRAM GPU or SFF UMA box, a whole rack, and a whole data center seems to be falling far short of what we might have imagined just a few years ago. The conversations I've heard have largely turned away from breathless anticipation of the next frontier model and toward attempts at hard-nosed evaluation of which tokens are worth the cost.
- eru 4d agoMaybe, though that's another kind of progress in itself. Very impressive progress!
- antupis 4d agoI think it’s more that pushing frontier is extremely costly and there is no free lunches in same way as 2024.
- int_19h 3d agoThe difference is still vast, it's just that the smaller stuff is "good enough" for many things now.
- whateveracct 4d agoastra is more parlor tricks than real gains tbh i swear they trained in on threejs in particular so those idiots on twitter could spam their garbage demos
- sampullman 4d agoI'm sure that's part of it, but I run it side by side in my review bot, and Astra medium effort consistently catches more issues than Sol 5.6, using fewer tokens. For coding it's a little harder to tell, but at least the prose feels a little better.
- int_19h 3d agoI strongly disagree. I'm working on a semantic model for Lojban, heavily AI assisted, using multiple models. I have basically all popular frontier models doing research and panel debates. Astra and Fable are both noticeably ahead of everything else including their previous iterations. When it comes to reviews, they can also find more issues in others (or even their own) code.
- f4dd 4d agoThey've not plataued but they're certainly not as impressive as the hype would have them to be. The reality is, it doesnt matter if LLMs keep getting more powerful because they still need a human to steer it. Without the human providing inputs to the LLM it just sits there and does nothing.
- arcanemachiner 4d agoYou don't need human input. Any coherent input will do the trick. You can, for example, hook it up to a logging system and have it fix errors as they occur on your platform.
- paulhebert 4d agoHave you tried this? How did it go? I’d be curious about: - your setup. How it all works - The types of errors it fixed and how quickly - Any regressions or issues it caused - The cost Thanks!
- int_19h 3d agoI have something like that running locally for my agentic harness (which includes cross-model messaging). There's a dedicated "product manager" session for it, and all other PMs are instructed to report issues with the harness as they occur to that session, while it is tasked to automatically prioritize and address them and coordinate fix deployment with other running sessions. It works surprisingly well. The errors fixed are both genuine errors in the harness itself, but increasingly so upstream bugs (in the underlying agent apps like Codex, or in Herdr, which is used to expose uniform programmatic access to all those different apps) for which it needs to come up with workarounds. No regressions so far. The cost is hard to judge on a subscription, especially when you're running really heavy tasks otherwise that dwarf any harness work.
- dismalaf 4d agoImpressed with the model performance or the chatbot/agent performance?
- rtpg 4d agoI'll take the opposite here. If someone put in frontier AI models from like .... last june I guess? in a box and let me run it with "decent" token throughput I would be happy. I think it's worth acknowledging that the power of LLMs at this point is not really so much in the smarts, but in the coordination and the surrounding harness tech. "Written english" turning into sequences of commands[0]. The whole agentic "stuff" in general. Tools + coordination is the superpower. The reasoning... it doesn't have to be _that_ good for the rest of the stuff to work. On good codebases and infra, at least. And I say this as someone who really would rather most of this stuff disappear! [0]: programming is obviously text to commands, but there's a loooooooot of futziness that LLM reasoning has let us remove in some flows
- anon373839 4d ago> If someone put in frontier AI models from like .... last june I guess? in a box and let me run it with "decent" token throughput I would be happy. You can have that! Qwen 3.8 Flash-Next is ~Opus 4.6 and runs nicely on a DGX Spark. And that’s just an architecture preview. The Qwen 4 family is expected to arrive this fall.
- rtpg 4d agoDGX Spark is a biiiiit costly but neat to hear! Do you know what kinda throughput you’re getting on that kinda setup? (I have a secondary problem of being “locked into” Claude Code by it being good enough for me, I’d probably need to investigate the other harnesses… my impression is other harnesses are a bit more aggressively OK with nuking your setup from orbit)
- anon373839 4d agoIt is costly, especially right now. I don’t think you can make a case for it on cost savings! The throughput in a single stream is about 50 tokens/sec (a bit less for prose, a bit more for code due to speculative draft acceptance rates) and about 2,000 tokens/sec for prefill. Both numbers are flat and stable as context accumulates. That’s what finally tilted me away from the Mac Studio despite its much superior memory bandwidth. I think these numbers may improve because the model is pretty new and optimizations aren’t done.
- octoberfranklin 4d agoPeople are definitely finding new things to use the models for, and orchestrating increasingly large swarms of agents in useful ways -- every single day, especially the last few months. But the basic single-NN frontier capability has been pretty stationary since Opus 4.8. Kimi K3 is almost as good as that with open weights, which has the frontier labs terrified. The only big thing on the horizon is if we can get diffusion models working reliably; that would be a big step forward. Inception's Mercury is AFAICT the leader here. It's stupifyingly fast but has obedience/hallucination problems that the autoregressives solved ~2 years ago. So it's not ready yet but improving. Also, FFS why is Grok the only model that knows how to do parallel tool calls? Such a useful ability and nobody else trains it in. Or if they do it just doesn't work.
- jandrese 4d agoI think we are in the second knee of the S curve, simply because we are hitting the point where is not enough hardware in the world to throw at this problem. These AI companies have bought everything they can and yet the models keep growing.