5 ms·
Didn't expect it to beat 3.8 27B so cleanly. Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy.
by rohansood15 21d ago
Didn't expect it to beat 3.8 27B so cleanly.
Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy.
- user43928 21d agoFor comparison with hosted models, GPT 5.6 Luna scores 67% on DeepSWE, compared to 59% here for Qwen. Luna is $0.20 / $1.20 vs $0.16 / $0.47 with Qwen.
- rohansood15 21d agoThis is a good counter argument. But you have to note that this is after OpenAI cut Luna costs by 80%. If you compare launch pricing, Qwen probably comes out ahead on a cost-performance basis.
- jrflo 21d agoThe luna cost cuts were real though, not a one time promotion or something, due to some optimization (probably distillation?) that openai did.
- QwenGlazer9000 21d agoWas it? Given the timing, I think they A. shat their pants since Deepseek flash just came out with insane pricing before the price hikes, and B. Anthropic is really struggling in model tiers below opus. It was smart for them to cut prices regardless of whether they had 80% efficiency gains or not
- throwaw12 21d agowhat if it was because of quantization and they haven't released the new benchmarks for it? Anything which changes the model needs new benchmarks I guess to compare with other models, otherwise you can benchmark Fable, and distill it to student model and keep claiming this is the Fable model
- dannyw 21d agoARC Prize has retested Luna after the discount and validated identical performance. (Also, quantization isn't inherently bad or damaging when done properly, e.g. QAT). These APIs are used heavily by enterprises at scale; with lots of performance telemetry, live evals, etc. You can't really silently nerf API models at scale without people noticing. Of course, what I said doesn't apply to non-API consumer sub models; there's many documented and officially confirmed instances of under-the-hood "juice/effort" adjustments. (Juice = a number your effort tier maps to underneath the hood; much like Inkling's effort=0.00 to 0.99).
- mattalex 21d agoYou assume that openai's inference is profitable and that they aren't just trying to bolster revenue before their IPO. The only indication that openai is profitable comes from openai (whom I wouldn't trust with any statement, especially when it comes to profitability). In fact there is evidence that inference is not profitable simply because the rate of losses doesn't seem to reduce as revenue increases: if inference had great margins, we would expect that as revenues increase, the amount of spend on training reduces as a fraction of total expenses. Since the loss-making fixed costs shrink as a fraction compared to the profitable inference, we should expect profitability to rise with total revenue. However, all leaks of openai's numbers seem to suggest the opposite: as revenues increase so do the losses.
- hluska 21d agoI don’t pay OpenAI’s bills - I pay what they charge me. Their cost accounting isn’t relevant to a user.
- aaa_aaa 21d agoArgument was that open ai cannot be profitable with this. But sure, use it while you can.
- mediaman 21d agoYou can make the other argument that China subsidizes the price and that they can't be profitable at this pricing level. From an industrial strategy standpoint, they already do this for many other industries with huge subsidized state loans. So we can go round and round on this, each with our made-up objections about how it's temporary or unrealistic or impossible or whatever, or we can just accept the prices as listed and use that to guide our economic decisions.
- aaa_aaa 21d agoPrivate companies cannot play that game too long. Profit from current state of AI is a mirage and sooner or later stuff will hit the fan.
- Almondsetat 21d ago>If you compare launch pricing Why?
- nl 21d agoWhy would anyone car what the launch price is? Comparing launch pricing is just an odd thing to do.
- rohansood15 21d agoBecause labs can learn to optimize inference post launch, plus can move to use bigger/better clusters depending on demand. It is not impossible to imagine Qwen cuts prices further with QAT/MTP-like improvements.
- nl 20d agoOr they could move from highly subsidized models like the Deepseek 4 launch pricing. Launch price is just like any other price. It's just a price. It's impossible to guess what might or might not happen. Compare the price now.
- criley2 21d agoThose prices are just tokens? Since each model uses different amounts of tokens to do the same thing, it's a misleading price that often makes open-weights look more competitive than they are, since most open weights models use dramatically more tokens and time to complete tasks than many frontier models. In Artifical Analysis's cost per task, Luna(max) costs $0.05 per task, and Qwen 3.8 27B costs $0.25 per task, a 5X increase. We'll see how 3.8-flash-next does.
- hadlock 21d agothe important thing is that Qwen 3.7 27B will run unlimited jobs on my consumer grade laptop at 60 tokens/second for free, forever, in about 1-2 years
- villish 21d agoThats only important if running it locally is critical for privacy reasons or just as a hobby. Time has a cost in business. If a model needs 30 million tokens to achieve a similar result as another that can do it in 10 million, that 60 tokens per second will take a long time.
- hadlock 21d agoRight now qwen 3.6 35b-a3b has a success rate of 92% and qwen 3.8 27b has a success rate of 96%. But the 35b moe does about 1080 tokens/s at concurrency 54, vs 480 tokens/s at concurrency 28. For our specific workflow on blackwell. Of course enormous batch jobs are different. I was explicit when I said consumer laptop.
- criley2 21d agoIt's not free. You're paying electricity and you're ignoring the cost of the hardware. Even on electricity alone, there are cloud providers who may beat your laptop on price per million tokens. Qwen 3.8 flash is interesting in this space. Not to say that there aren't other benefits of running models locally, I loaded Qwen 3.8 27B 6bit MLX just yesterday.
- claudeIsDown 21d agoSounds like discrete propaganda
- Squarex 21d agoI don't like these comparisons. Sure it is impressive, but it does not have a world knowledge of larger models. It has most of theirs intelligence.
- LaurensBER 21d agoIf/when we can get larger context this will mostly be mitigated by these smaller models being able to search the internet. Self-learning/improving would be even better but that's still a long way to go.
- redox99 21d agoSearch results suck because the web sucks these days. The big models from OpenAI/Anthropic have every book in existence baked into them
- jameshart 21d agoI don’t think that’s the right way to think about LLM ‘knowledge’. They don’t have absolute recall of everything in the training set. They have been trained so that they have weights that can predict what those books might say - that is, if they read them they would find the contents unsurprising. That doesn’t mean it wouldn’t be helpful to pull relevant passages of text directly into context for a particular task.
- infp_arborist 21d agoDoes it really matter? What about including all relevant and up-to-date literature as skills for local models? I have no experience with this but I am pretty sure someone has already thought about it.
- rohansood15 21d agoFor world knowledge, you'd want it to find and reference the source material to be sure. At that point, it doesn't matter if the knowledge is embedded.
- RobertasTa 21d ago[flagged]
- gruez 21d ago>Opus 4.6 Max self-hosted at 30 tok/s on a 5k Macbook in Aug 2026. The LLM timelines are crazy. How much memory does this translate to and what quantization (if any) were applied?
- rohansood15 21d ago128GB, 4-bit quantized.
- dist-epoch 21d agoIt's a much bigger model, with a next-gen architecture. It's expected to be much better.
- hedora 21d agoMy AMD strix halo box (haven’t benchmarked yet) should also run it reasonably well. It was $1400 at launch, and is $4K now. Your mac is < $2K in Biden-era dollars. Presumably the economy will eventually recover; maybe in one Moore’s law doubling if the midterms go outrageously well. That’ll be two doublings since the halo launched. I’d expect this model to run on a sub $1K box by then. $2K ought to get you a 512b parameter model at that point. If we have to wait out the rest of the term, the cost cliff will be even more pronounced when it hits.
- aftbit 21d agoI believe you're underestimating the lag inherent in the economy. Even if we grant the idea that the political party controlling the US House/Senate has a significant impact on the economy, and that the current party is BAD and the next one would be GOOD, I would still expect that things will continue getting WORSE for a good 4 to 8 years before they get better again. And that's even with assuming that we can continue to ignore the long-term problems like social security insolvency, the debt bomb, or climate change forever.
- NewJazz 21d agoYou know the memory cartel isn't even close to being broken, right?
- overgard 21d agoCurious, how are you running it and what quantization are you using? I've mostly been using MTPLX; 125B sort of looks like it'd be right at the limits of my 128GB MacBook once you factor in KV cache and context window.. wondering if it's worth it compared to the 27B model which gives me a lot of headroom or even a 72B model.