6 ms·
We made something called Divergence-300 @32 (and later @512) which tests actual inference across 32 tokens on a held out test (Terminal Bench, DeepSWE, Math etc
by danielhanchen 28d ago
We made something called Divergence-300 @32 (and later @512) which tests actual inference across 32 tokens on a held out test (Terminal Bench, DeepSWE, Math etc)
We do plan to do larger benchmark suites though!
- johndough 28d agoGreat to hear that you are planning larger benchmarks! I am particularly interested in longer-running tasks with many steps and self-correction. Divergence is fine as long as the model can still solve the task, which Divergence-300 @32 does not measure. The current benchmark suites that frontier AI labs use are probably a good fit, e.g. https://z.ai/blog/glm-5.3#:~:text=Performance%20across%20comparison%20models https://z.ai/blog/glm-5.3#:~:text=Performance%20across%20com... https://www.kimi.ai/ai-models/kimi-k3#:~:text=Performance%20at%20the%20frontier https://www.kimi.ai/ai-models/kimi-k3#:~:text=Performance%20... https://www.anthropic.com/news/claude-opus-5 https://www.anthropic.com/news/claude-opus-5 https://openai.com/index/gpt-5-6/ https://openai.com/index/gpt-5-6/ But guessing from your current benchmarks, I assume that you are severely compute-constrained. What is your time budget?