9 ms·
Beats Opus 4.7 Max (w/ Claude Code) on DeepSWE (42.2 vs 40). Looks like Qwen's 27B models continue to pack some punch. Unsloth's GGUF quants are up: https://hu
by scrlk 1mo ago
Beats Opus 4.7 Max (w/ Claude Code) on DeepSWE (42.2 vs 40). Looks like Qwen's 27B models continue to pack some punch.
Unsloth's GGUF quants are up: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF https://huggingface.co/unsloth/Qwen3.8-27B-GGUF
- edg5000 1mo agoThat's crazy, considering the massive size difference. But the small Qwen models are known for punching above their weight.
- nblgbg 1mo agoIs there any advantage to using the model from Unsloth compared with https://huggingface.co/Qwen/Qwen3.8-27B-FP8 https://huggingface.co/Qwen/Qwen3.8-27B-FP8 ?
- benxh 1mo agoDepends on what software/hardware you'll run it. GGUFs from Unsloth can run on pretty much every single potato; full weights need beefy gpus
- petu 1mo agoUnsloth one is gguf for llama.cpp (and some other on-device engines). So advantage is not having to produce your own quantisation / gguf from .safetensors you've linked.
- 4chandaily 1mo agoRun the unsloth if you are using llama.cpp (GGUF) Run the one you linked if you are running vllm (safetensors)
- danielhanchen 1mo agoWe also made NVFP4 ones if that helps! https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4 https://huggingface.co/unsloth/Qwen3.8-27B-NVFP4
- hadlock 1mo agoThis is the version we'll be testing on our rtx 6000 today! Thank you
- eblanshey 1mo agoWhy not just run FP8 on vLLM with that much vRAM? It's plenty fast.
- hadlock 1mo agoFor high concurrency, using the blackwell's native native W4A4 MLP compute path, nvfp4 is something like a 1.2-1.5x performance increase over FP8. We're doing data enrichment (so, tasks completed successfully + tokens/second) so the performance bump shows up in the tasks/month number. I am just now getting the benchmarks running against 3.8 27b but I expect similar results from benching 3.6 27b at the same quant.
- eblanshey 1mo agoI see. Did you see any intelligence degradation between FP8 and NVFP4 for 3.6 27B? You're using vLLM, right?
- hadlock 1mo agoI didn't have time to run this as well, but we're getting a 99% agent completion rate across all tasks and 98% task decision that matches the human selected option(s) on Qwen 3.8 27B @ NVFP4. There may be a difference between FP8 and NVFP4 but it's inconsequential for our data enrichment purposes.
- satvikpendem 1mo agoUnsloth usually also fixes the models when they bork something, which always happens. For Gemma for example the tool calling wasn't working for the longest time.
- danielhanchen 1mo agoThat wasn't our problem right? Gemma officially updated tool calling which we adopted
- ycui7 1mo agoif you have the VRAM, use offical release. quantized model lose focus after long context and can do damages or thinking loop
- WithinReason 1mo agoI wish each quant was benchmarked on the same tests as the original network so we could compare their performance
- scrlk 1mo agoUnsloth publishes KL divergence numbers which measures how much the quantised probability distribution changes vs unquantised: https://unsloth.ai/docs/models/qwen3.8#quantization-analysis https://unsloth.ai/docs/models/qwen3.8#quantization-analysis It's a bit bare at the moment, I assume they are going to add further detail later (eg comparison to other quants), similar to their other releases.
- zargon 1mo agoKL divergence is nothing close to a replacement for benchmarks. As flawed as benchmarks are, KL divergence is a barely useful signal. The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is.
- lostmsu 1mo ago> The fact that Unsloth only just started publishing KL divergences shows how unserious the quantization space is. Just wanted to say that this is a very important point that I totally agree with. People are obsessed with KL divergence, but it is yet to be demonstrated to be a descent proxy for agentic coding benchmarks.
- nullc 1mo agoIt's a cheap to evaluate proxy for totally broken or not, which is a good start. It also has a lot of resolution and not a lot of noise. Better would be multi-turn benchmarks with tools but getting good precision and accuracy for that is hard and computationally expensive.
- lostmsu 1mo ago
- NitpickLawyer 1mo ago> Beats Opus 4.7 Max I'm a huge open model fan, and have used them since forever, even have daily drivers for on-prem dev, but no. They do not beat opus on real-world usage. Qwen models are impressively good for what they are, are "good enough" for plenty tasks, can be ran locally on decently priced hardware, and so on. They certainly have their uses, and the field in general has advanced faster than my early expectations. But to compare a 27B model to SotA behemoths from a few months ago is doing everyone a disservice, especially people who pick it up, try to use them just like API models, and leave disappointed and confused. Number goes up on a benchmark isn't it.
- KronisLV 1mo ago> ...but no. They do not beat opus on real-world usage. I agree, but then we just need meaningful benchmarks that clearly show that! Otherwise it's hand waving about something that should be put on paper in quantifiable terms.
- bewareofscams 1mo agoOnly useful benchmarks are those you (in particular) don't have access to.
- rhdunn 1mo agoThe only useful benchmarks are those you've created for your specific workflow. Only then can you assess whether a given model is better or worse for what you are using it for. There are tools like promptfoo designed for this.
- niek_pas 1mo agoA wise man once said, "not everything that counts can be counted, and not everything that can be counted counts".
- valvar 1mo agoThat’s well and good, but how are we supposed to evaluate the accuracy of random HN comments without anything resembling somewhat objective metrics? People say all manner of things, and usually it’s contradictory. What heuristic do you propose?
- Foobar8568 1mo agoConsidering the clusterfuck that is opus 5 or even fable, if Qwen 27B is trully better than Opus 4.7 Max, I will rejoice.
- UncleOxidant 1mo agoIf it's as good as Sonnet 4.6 for most things I'd be happy.
- ranguna 1mo agoSame
- ferrouswheel 1mo agoYeah Opus 5 is almost unusable as a daily driver without making me go insane from excessive claude babble.
- UncleOxidant 1mo agoGood morning Dario!
- jrflo 1mo agoThat kind of result makes me suspicious of benchmaxxing. Qwen 27B is 100x smaller than Opus 4.7. Is it really 100x more parameter-efficient? Two orders of magnitude is hard to believe. I don't have the hardware to run a 27B, but I'm curious what real world use is like. Maybe I'll have to buy some usage on a cloud provider to run my own tests, but this seems fishy to me.
- CuriouslyC 1mo agoQwen small models are heavily coding focused, whereas Opus is everything to everybody (even if code is their bread and butter). The downside is they'll frequently hallucinate world knowledge so they need to be RL'd to double check their knowledge against sources and verify facts/library names/etc.
- dannyw 1mo agoIt's very agentic coding focused; and I'd say a good executor but certainly not Opus in scale; overall knowledge; long-horizon work and recovery; etc. e.g. If you try to chat to it about something philosophical for example, or maybe a debate / creative writing, then you'll very quickly see how it is still a much smaller model at the end of the day. Still, it's such a relatively accessible model to run, and I find a big part of leveraging smaller models is to give it well-scoped tasks; not too high level or ambitious ones. Very impressive for its size and the ability to run locally :)
- jrflo 1mo agoInteresting! I'll definitely give that a shot then.
- Aurornis 1mo agoIn the local LLM communities there is a lot of respect for the Qwen models, but everyone comes to acknowledge that they do a lot of benchmaxxing after using them. Even at full precision they're never as good as models with similar benchmarks.
- alentred 1mo agoOh, goody goody goody! Can't wait for a Q4M with *MTP* support. Does anyone know if there are plans for it? I am currently using Qwen 3.6 on RTX 3090 and I have to admit that without MTP it would be too slow to be acceptable for me (30-35 tok/sec without MTP, 60-70 with MTP). Without MTP I would just use OpenRouter and rather pay for speed despite having a capable local setup.