5 ms·

5.6 Sol (max) being cheaper than all of these is wild, considering how good the output is too

by midnightbobarun 2mo ago

giancarlostoro 2mo ago
Probably because they made ASICs to run inference for less.
- brookst 2mo ago
  Are those actually deployed at scale yet?
  - brcmthrowaway 2mo ago
    Yes.
    - wmf 2mo ago
      I hate to disagree with Broadcom Throwaway himself but it's unlikely that the OpenAI Jalapeno ASIC has been deployed yet. It takes 6-12 months to test, develop software, ramp production, etc.
Schiendelman 2mo ago
This must be on API costs, not counting the $100/200 tiers, right?
- anuramat 2mo ago
  yes; fyi usage limits on the $200 claude sub correspond to at least $1.2k/week in api tokens
  - Schiendelman 2mo ago
    More than that using Fable!
nijave 2mo ago
I think on swebench verified luna was only like 3% points lower for 1/5 the cost Like 96% vs 93% or something
- mdgld 2mo ago
  Yeah, sol is impressive but IMO Luna is the real standout (and terra is the laggard of the group) for performance/cost
- twotwotwo 2mo ago
  There is a blog post waiting to be written (that I won't write) about the size/effort tradeoffs, and particularly how small models get some surprisingly good results with lots of turns and reasoning. DeepSWE will let you chart turns taken or tokens used, and FrontierCode will chart tokens. If you use that, you can see Sol high and Terra max get about the same DeepSWE number, but Terra max takes twice the turns. Luna max scores a smidgen lower with even more turns. Smaller models relying on lots reasoning may "scale down" better on easier tasks, because unlike size, reasoning effort is dynamic: the model can see the task looks easy and stop. On DeepSWE, the cost curves for the three 5.6 models are almost on top of each other, but on FrontierCode Extended, the version of FrontierCode with the most everyday tasks in the mix, there's a spread of costs at the ~55% level. The recent Laguna S 2.1 model (118B, 8B active) puts up surprising coding numbers for its size, and the lab behind it specifically credits its "way of working (persistence, verification, willingness to backtrack)". Some other open models that folks report getting good mileage out of seem to get there partly by throwing a lot of reasoning at the problem. There is a little bit of a question, if some models rely on getting it wrong a bit more at first and external checks catching the problems, of whether they're also more frequently getting things wrong they can't self-verify (say, quality of UI or API design) and then it falls to the human to find it. Still, getting the results they're getting at all is neat. Some benchmarks historically favored reporting only on the max variants, maybe because they want to show the frontier? but that is not always what you need for practical decision. (AA has the full effort sweep for Opus 5 and Sol/Luna/Terra at least.) And at least FrontierCode finds Opus 5 taking a hit in performance above 'medium'. I am not trying to pick a winner here. I'm probably not going to use tiny models on max for everything, but I think it's cool that you can get so much more out of a small model by amping up reasoning, tool use, and persistence.
impulser_ 2mo ago
It shouldn't be surprising OpenAI does have the most compute out of all the major labs. The only reason why Anthropic models are expensive is they are the most in demand models in the world and Anthropic is fighting for compute. The only way to you limit demand for your model is increasing API pricing this is also why Anthropic probably has great margin and probably is profitable compared to OpenAI.
- scrlk 2mo ago
  Not just compute for OAI, GPT-5.6 is more token efficient across the board vs the Anthropic equivalents: https://artificialanalysis.ai/models?intelligence-index-token-use=intelligence-vs-output-tokens-per-task&cost=intelligence-vs-cost-per-task&total-cost=intelligence-vs-total-cost#token-use https://artificialanalysis.ai/models?intelligence-index-toke... No wonder why Tibo can afford to hit the reset button liberally.
- charcircuit 2mo ago
  I also suspect there is a price fixing agreement between all of the inference providers for Claude (such as Amazon, Anthropic, Microsoft, etc).
  - wmf 2mo ago
    "Price fixing" isn't the correct term here but yes, it's very common to have the same price across different retailers/resellers.
    - charcircuit 2mo ago
      There is a difference between the market discovering a price and a bunch of retailers/resellers entering an agreement to sell at a specific price.
  - vikramkr 2mo ago
    I doubt there's any sort of criminal behavior there - the model is anthropic's up and anthropic probably charges a very expensive license fee that's the same for all of them, and their cogs on compute aren't going to be wildly different, so the main drivers of the cost are roughly the same and they're all offering customers the same end product so the prices would likely also be similar in the end
deleted 2mo ago
[deleted]

5.6 Sol (max) being cheaper than all of these is wild, considering how good the output is too

giancarlostoro 2mo ago

Probably because they made ASICs to run inference for less.

brookst 2mo ago
Are those actually deployed at scale yet?
- brcmthrowaway 2mo ago
  Yes.
  - wmf 2mo ago
    I hate to disagree with Broadcom Throwaway himself but it's unlikely that the OpenAI Jalapeno ASIC has been deployed yet. It takes 6-12 months to test, develop software, ramp production, etc.

Schiendelman 2mo ago

This must be on API costs, not counting the $100/200 tiers, right?

anuramat 2mo ago
yes; fyi usage limits on the $200 claude sub correspond to at least $1.2k/week in api tokens
- Schiendelman 2mo ago
  More than that using Fable!

nijave 2mo ago

I think on swebench verified luna was only like 3% points lower for 1/5 the cost Like 96% vs 93% or something

mdgld 2mo ago
Yeah, sol is impressive but IMO Luna is the real standout (and terra is the laggard of the group) for performance/cost
twotwotwo 2mo ago
There is a blog post waiting to be written (that I won't write) about the size/effort tradeoffs, and particularly how small models get some surprisingly good results with lots of turns and reasoning. DeepSWE will let you chart turns taken or tokens used, and FrontierCode will chart tokens. If you use that, you can see Sol high and Terra max get about the same DeepSWE number, but Terra max takes twice the turns. Luna max scores a smidgen lower with even more turns. Smaller models relying on lots reasoning may "scale down" better on easier tasks, because unlike size, reasoning effort is dynamic: the model can see the task looks easy and stop. On DeepSWE, the cost curves for the three 5.6 models are almost on top of each other, but on FrontierCode Extended, the version of FrontierCode with the most everyday tasks in the mix, there's a spread of costs at the ~55% level. The recent Laguna S 2.1 model (118B, 8B active) puts up surprising coding numbers for its size, and the lab behind it specifically credits its "way of working (persistence, verification, willingness to backtrack)". Some other open models that folks report getting good mileage out of seem to get there partly by throwing a lot of reasoning at the problem. There is a little bit of a question, if some models rely on getting it wrong a bit more at first and external checks catching the problems, of whether they're also more frequently getting things wrong they can't self-verify (say, quality of UI or API design) and then it falls to the human to find it. Still, getting the results they're getting at all is neat. Some benchmarks historically favored reporting only on the max variants, maybe because they want to show the frontier? but that is not always what you need for practical decision. (AA has the full effort sweep for Opus 5 and Sol/Luna/Terra at least.) And at least FrontierCode finds Opus 5 taking a hit in performance above 'medium'. I am not trying to pick a winner here. I'm probably not going to use tiny models on max for everything, but I think it's cool that you can get so much more out of a small model by amping up reasoning, tool use, and persistence.

impulser_ 2mo ago

It shouldn't be surprising OpenAI does have the most compute out of all the major labs. The only reason why Anthropic models are expensive is they are the most in demand models in the world and Anthropic is fighting for compute. The only way to you limit demand for your model is increasing API pricing this is also why Anthropic probably has great margin and probably is profitable compared to OpenAI.

scrlk 2mo ago
Not just compute for OAI, GPT-5.6 is more token efficient across the board vs the Anthropic equivalents: https://artificialanalysis.ai/models?intelligence-index-token-use=intelligence-vs-output-tokens-per-task&cost=intelligence-vs-cost-per-task&total-cost=intelligence-vs-total-cost#token-use https://artificialanalysis.ai/models?intelligence-index-toke... No wonder why Tibo can afford to hit the reset button liberally.
charcircuit 2mo ago
I also suspect there is a price fixing agreement between all of the inference providers for Claude (such as Amazon, Anthropic, Microsoft, etc).
- wmf 2mo ago
  "Price fixing" isn't the correct term here but yes, it's very common to have the same price across different retailers/resellers.
  - charcircuit 2mo ago
    There is a difference between the market discovering a price and a bunch of retailers/resellers entering an agreement to sell at a specific price.
- vikramkr 2mo ago
  I doubt there's any sort of criminal behavior there - the model is anthropic's up and anthropic probably charges a very expensive license fee that's the same for all of them, and their cogs on compute aren't going to be wildly different, so the main drivers of the cost are roughly the same and they're all offering customers the same end product so the prices would likely also be similar in the end

deleted 2mo ago

[deleted]