7 ms·
Are there benchmarks for the various Qwen3.8-27B quants that actually measure writing code, maybe even with multiple steps? Low KL divergence does not mean much
by johndough 29d ago
Are there benchmarks for the various Qwen3.8-27B quants that actually measure writing code, maybe even with multiple steps? Low KL divergence does not mean much when the model gets stuck in doom loops all the time.
I could of course download and test myself, but that would take days with my internet connection.
- InvertedRhodium 29d agoI tested Qwen 3.8 on the Blade CTF last night, it took 3 hours but got the correct answer. I know that didn’t answer your question but I was looking for a test suite and couldn’t find anything. After reading the logs, there is far less doom looping than with 3.6, but whether that’s a one off or not is up for debate. Q4_K_P
- Balinares 29d agoI anecdotally observed the same. Interestingly, it also seems to tend toward self-correcting, which makes lower quantizations borderline usable. There'll be more faffing around, but still converging toward a solution. I wonder if that's a deliberate product of its RL.
- danielhanchen 29d agoWe made something called Divergence-300 @32 (and later @512) which tests actual inference across 32 tokens on a held out test (Terminal Bench, DeepSWE, Math etc) We do plan to do larger benchmark suites though!
- johndough 28d agoGreat to hear that you are planning larger benchmarks! I am particularly interested in longer-running tasks with many steps and self-correction. Divergence is fine as long as the model can still solve the task, which Divergence-300 @32 does not measure. The current benchmark suites that frontier AI labs use are probably a good fit, e.g. https://z.ai/blog/glm-5.3#:~:text=Performance%20across%20comparison%20models https://z.ai/blog/glm-5.3#:~:text=Performance%20across%20com... https://www.kimi.ai/ai-models/kimi-k3#:~:text=Performance%20at%20the%20frontier https://www.kimi.ai/ai-models/kimi-k3#:~:text=Performance%20... https://www.anthropic.com/news/claude-opus-5 https://www.anthropic.com/news/claude-opus-5 https://openai.com/index/gpt-5-6/ https://openai.com/index/gpt-5-6/ But guessing from your current benchmarks, I assume that you are severely compute-constrained. What is your time budget?
- zdragnar 29d agoPurely an anecdote, but I've found Qwen3.8-27b doesn't doom loop like previous Qwen models would. With that said, it absolutely thinks in circles- it'll prepare to do something, say it is now ready to do it, then follow that with three paragraphs that all start with Acutally... Oh wait, I should check first... Hmm, hmm... I should stop guessing and just do it. Okay, I'm ready to do the thing now... Actually, wait... It takes forever, but it does actually get around to making things work, and it is more thorough and produces better code than previous qwen models. You just need to let it run quite awhile.
- zenoprax 28d agoI've seen the same thing. I tried the "superpowers" meta-harness and gave it a simple web app task and it spent 4 hours to make a basic timer app. I might try restricting the amount of thinking it is allowed to do to 500-1000 tokens.
- stkdump 28d agoThere is a native reasoning effort setting. It defaults to xhigh, I guess to get the best benchmark results, but you can just run it on medium or low instead, or for simple things even disable thinking outright.
- lifepillar 28d agoAccording to this guy [0], medium is the level that tends to produce way less tokens in agentic workflows ("low" may output less per response, but then the model makes more mistakes, so it needs to iterate more). [0] https://m.youtube.com/watch?v=z64J6bC16iQ https://m.youtube.com/watch?v=z64J6bC16iQ
- zenoprax 28d agoGreat, thanks! I'll try it out.