7 ms·
It's the same story every time OpenAI or Anthropic releases a new model. They are generous with compute for the first few days, and use maximum fidelity with un
by cainxinth 7d ago
It's the same story every time OpenAI or Anthropic releases a new model. They are generous with compute for the first few days, and use maximum fidelity with uncompressed weights. Everything runs at its best to make a good first impression. But eventually they pare things back and the models perform a little worse.
- blurbleblurble 7d agoOr a lot worse
- holler 7d agoso, AGI is cancelled?
- __MatrixMan__ 7d agoAGI for the peasants is cancelled.
- elwell 7d agoTrogdor - the AGInator
- baby 7d agoYou think they introduce stronger quantization after a few days?
- boredatoms 7d agoFor sure they quickly move to q8, the output quality difference to bf16 is small compared to the speed/capacity gain
- NineStarPoint 7d agoYeah q8 made so littler difference back when I was testing such things I'd be surprised if people could quickly notice that as a change. It's got to be either further quantized or some other type of optimization that kicks in when people notice the drop.
- selectodude 7d agoNVFP4 would buy them a huge increase in capacity but I think it would be noticeable.
- Caracas288 7d agoWhy doesn't someone just try to measure this next time!?
- embedding-shape 7d agoCan't really measure without being sure you aren't being messed around with, when it's a remote platform. Stupidly easy to detect when people run such benchmarks/tests against you as well.
- nonethewiser 7d agoCould this explain Opus?
- torginus 7d agoSome people here have remarked previously that while reduced precision doesn't show up in quick prompts, it does severely impact these models' ability to perform long running tasks - to the point that running these big models with severe quantization might be counterproductive as smaller but less quantized ones perform better.
- dooglius 7d agoDo you have hard evidence of this assertion?
- simlevesque 7d agoWe can't have hard evidence. It's a SaaS and they own the code and the machine it runs on. So it may be a widespread hallucination. But there's no evidence of that either.
- fragmede 7d agoWe could still have soft evidence though. Make a Todo app on Monday, and make a Todo app on Tuesday, and see what it makes in comparison.
- marcus_cemes 7d agoYou would need a significant sample size to make any sort of conclusion from such a probabilistic process. Then there's the issue of how you would actually grade/compare.
- ArvidSu 7d agoYou only need to come up with a catchy "SomethingBench" name, post it on reddit/x and now you're an ai sage. Not to disparage the launch/after comparison though, I'd genuinely enjoy a data point like that
- luckydata 7d agosomeone already does that https://aistupidlevel.info/ https://aistupidlevel.info/
- chaimtweiss 7d agoIt's actually a extremely cool site, and fascinating to view the results off the AI bots i use.
- dooglius 7d ago
- Vetch 7d agoThe most charitable explanation I can think of for this is something like regression to the mean. When a model is first released, there'll be a subset of users who, just by chance, sample the highest quality band of the distribution that answers their query. Some of them will rush over to social media and post about how amazing a model is. Over time, those users' mental model of responses will converge but they'll perceive the model's return to typical performance as a downgrade. This guess/explanation predicts that most users won't match what the initial social media hype claims, doesn't discount user experience as simple habituation nor does it assume companies are lying when they say there have been no changes to the model itself (quantization included). I also think there's an aspect where initial testing is more forgiving because the more persnickety polish bits can be ignored and tests are likely to have similar structure to things that can be trained for. Meanwhile, actual specific work items are a broader unusual distribution with more stringent acceptance criteria. Personally, I can detect a separation between Sol and Astra (but not as large as that between Opus and Fable). While they can solve most of the same problems, Astra takes less time, is less frustrating to talk to, is cleaner, notices more, spins wheels less and requires less corrections.
- kmeisthax 7d agoTo add onto this, if you use a shiny new model and it gives you a turd, you're not going to tweet about it ("hey guys, look what I made with Astra! Nothing!"), and even if you do nobody is going to interact with it so it does poorly in the algorithm, because it has to compete with all the people using the new model to make something that looks impressive. Then people get tired of the magic trick and the logic flips.
- zaphirplane 7d agoReally? there would be complaints, it’s expensive and doesn’t do as well
- foolswisdom 7d agoWhen it's happened to me, I shrugged and went back to the way I did things before. Then again, I'm not a vocal social media user by any means.
- holoduke 7d agoI am sure every input send to openai is prechecked by a dumb model and then send to another one. They heavily tweak this to improve performance.
- OneOffAsk 7d agoIt’s all speculation (you too), but I think the effect you’re describing is instead getting calibrated to the model’s limits. Next time a new model comes out, wait a month before trying and see if you have the same feeling of rapid quality decline after a few days. I did after I jumped back into it mid 5.x or whatever ChatGPT after paternity leave. Blown away for a few days, worried about my job for a few days, then increasingly aware of its limits.
- cbg0 6d agoThis is a bit of an urban myth. There are trackers which keep historical performance and Sol hasn't been nerfed: https://marginlab.ai/trackers/codex/ https://marginlab.ai/trackers/codex/
- webern777 5d agoThe problem with this is I won't get any clicks posting that performance is the same and I don't let facts get in the way of more clicks.