6 ms·

This is really a great achievement: "Astra dominates the output token frontier" Many labs used increased thinking to boost benchmark scores and performance. Mo

by __jl__ 12d ago

This is really a great achievement: "Astra dominates the output token frontier" Many labs used increased thinking to boost benchmark scores and performance. Most of the Chinese models were doing that for a while. Google and Anthropic as well. Not OpenAI. 5.6 already was much more token efficient than other models and Astra beats Sol in token efficiency by a wide margin. Edit: Just to make the point: Astra (max) has the 2nd highest score and the third lowest output tokens (among the models shown by AA).

Scaevolus 12d ago
GPT-6 has a looped transformer / recurrent depth architecture, so it gets some internal CoT reasoning "for free" with no output tokens. This also makes it much harder to monitor its reasoning.
jsnell 12d ago
The apparent advantage is exaggerated by them running Astra at six different effort levels, and almost everything else at just the maximum available effort. I don't really understand why they keep doing this. Either run and report everyone at multiple effort levels, or run everyone at only one. But alsi, token efficiency seems pretty artificial? For example tokenizers are different from model to model. The cost/perf Pareto frontier seems a lot more meaningful (and Astra does very well at that too, just to be clear. It seems to be a great model.)
ssivark 12d ago
Duh, that's just a benchmarking artifact. If you run one model at 4x recurrence compared to another, you get 4x thinking without increasing the tokens. So of course it's going to dominate the perf at given output token level. The truly sensible comparison is perf -vs- thinking flops (because each model might be a different unknown size, but labs are very secretive about what they're actually running under the hood) or perhaps cost (which can be misleading because of subsidies, but is at least practically relevant in the moment).
- water-drummer 12d ago
  But the end user doesn't care about flops for closed models. All they care about is how much it ends up costing them.
  - ssivark 11d ago
    Which is why I said that cost might be a better metric than output tokens. But even that is somewhat misleading -- because there isn't really a single price -- there are a variety of offers / deals / subsidies, including subscription plans. But if we're actually talking benchmarking intelligence -- as evidenced by "Artificial Analysis Intelligence Index" and the rest of the reporting -- then comparing systems with different amounts of recurrence on output tokens is flawed.

by __jl__ 12d ago

Scaevolus 12d ago

GPT-6 has a looped transformer / recurrent depth architecture, so it gets some internal CoT reasoning "for free" with no output tokens. This also makes it much harder to monitor its reasoning.

jsnell 12d ago

The apparent advantage is exaggerated by them running Astra at six different effort levels, and almost everything else at just the maximum available effort. I don't really understand why they keep doing this. Either run and report everyone at multiple effort levels, or run everyone at only one. But alsi, token efficiency seems pretty artificial? For example tokenizers are different from model to model. The cost/perf Pareto frontier seems a lot more meaningful (and Astra does very well at that too, just to be clear. It seems to be a great model.)

ssivark 12d ago

Duh, that's just a benchmarking artifact. If you run one model at 4x recurrence compared to another, you get 4x thinking without increasing the tokens. So of course it's going to dominate the perf at given output token level. The truly sensible comparison is perf -vs- thinking flops (because each model might be a different unknown size, but labs are very secretive about what they're actually running under the hood) or perhaps cost (which can be misleading because of subsidies, but is at least practically relevant in the moment).

water-drummer 12d ago
But the end user doesn't care about flops for closed models. All they care about is how much it ends up costing them.
- ssivark 11d ago
  Which is why I said that cost might be a better metric than output tokens. But even that is somewhat misleading -- because there isn't really a single price -- there are a variety of offers / deals / subsidies, including subscription plans. But if we're actually talking benchmarking intelligence -- as evidenced by "Artificial Analysis Intelligence Index" and the rest of the reporting -- then comparing systems with different amounts of recurrence on output tokens is flawed.