8 ms·
Duh, that's just a benchmarking artifact. If you run one model at 4x recurrence compared to another, you get 4x thinking without increasing the tokens. So of co
by ssivark 14d ago
Duh, that's just a benchmarking artifact. If you run one model at 4x recurrence compared to another, you get 4x thinking without increasing the tokens. So of course it's going to dominate the perf at given output token level. The truly sensible comparison is perf -vs- thinking flops (because each model might be a different unknown size, but labs are very secretive about what they're actually running under the hood) or perhaps cost (which can be misleading because of subsidies, but is at least practically relevant in the moment).
- water-drummer 14d agoBut the end user doesn't care about flops for closed models. All they care about is how much it ends up costing them.
- ssivark 13d agoWhich is why I said that cost might be a better metric than output tokens. But even that is somewhat misleading -- because there isn't really a single price -- there are a variety of offers / deals / subsidies, including subscription plans. But if we're actually talking benchmarking intelligence -- as evidenced by "Artificial Analysis Intelligence Index" and the rest of the reporting -- then comparing systems with different amounts of recurrence on output tokens is flawed.