6 ms·

There are numerous benchmarks that measure cost per task, which factors out tokens entirely. Gemini 3.8 flash is significantly lower than Sol on basically all o

by gundmc 14d ago

There are numerous benchmarks that measure cost per task, which factors out tokens entirely. Gemini 3.8 flash is significantly lower than Sol on basically all of them https://artificialanalysis.ai/#cost-tabs https://artificialanalysis.ai/#cost-tabs That said, Luna is the undisputed king here at the moment and is what I use as my workhorse model.

NicoJuicy 14d ago
It's so funny how many people diverge on the same model. Ps. For the last week I diverged to Luna too, still need to check 3.8 flash. But 3.6 flash was my go-to model 3 weeks ago and before it was deepseek flash/pro for a while. None of the claude models seemed cost effective though.
criley2 14d ago
>There are numerous benchmarks that measure cost per task, which factors out tokens entirely. Gemini 3.8 flash is significantly lower than Sol on basically all of them https://artificialanalysis.ai/#cost-tabs https://artificialanalysis.ai/#cost-tabs Not sure if you read your own link but Sol 56 high ranks smack between Gemini 3.8 flash medium and high. Gemini 3.8 flash comes in as more expensive per task than Sol 56 high according to artificial analysis. Luna high is literally 30X cheaper than Gemini 3.8 flash high. You can limit the model viewer and they're getting better at testing multiple effort levels now: https://artificialanalysis.ai/?models=gpt-5-6-sol-medium%2Cgemini-3-8-flash-medium%2Cgemini-3-8-flash%2Cgpt-5-6-sol-high%2Cgpt-5-6-luna-high%2Cgpt-5-6-sol-xhigh#cost-tabs https://artificialanalysis.ai/?models=gpt-5-6-sol-medium%2Cg... One reason is clear: Sol uses dramatically fewer output tokens than Gemini 38 flash https://artificialanalysis.ai/?models=gemini-3-8-flash%2Cgemini-3-8-flash-medium%2Cgpt-5-6-sol-high%2Cgpt-5-6-sol-medium%2Cgpt-5-6-luna-high&cost=intelligence-vs-cost-per-task#output-tokens-tabs https://artificialanalysis.ai/?models=gemini-3-8-flash%2Cgem...
- PunchTornado 14d ago
  I open the link and I see Flash 3.8 high at 0.58 and Sol at 0.95. I don't understand why you say that "Sol 56 high ranks smack between Gemini 3.8 flash medium and high" but that is clearly wrong.
  - criley2 14d ago
    On cost per intelligence task, Gemini38flash and Sol56 trade back and forth on cost depending on effort level. https://i.imgur.com/zPaWPXx.png https://i.imgur.com/zPaWPXx.png As seen in this image, literally: Sol56 high ranks in between Gemini 38 medium and high. The image proves it. I also included Sol56 xhigh, which ranks above even Gemini38 high.
refactor_master 14d ago
I don't know if my code is just "complex", but I find that Luna on max ignores the surrounding style and completely ignores logical consequences of a change, like just writing `del arg1, del arg2, ...` instead of dropping it from the surrounding code. All LLMs make questionable decisions at times, but Luna requires so much guidance that it's faster to just type it out yourself. What kind of routine tasks can one accomplish with such a model?
- kasey_junk 14d ago
  Do you have code formatters, linters and static analysis? I can get extremely dumb models to get our code style correct because of those guard rails and a specific style document.

by gundmc 14d ago

NicoJuicy 14d ago

It's so funny how many people diverge on the same model. Ps. For the last week I diverged to Luna too, still need to check 3.8 flash. But 3.6 flash was my go-to model 3 weeks ago and before it was deepseek flash/pro for a while. None of the claude models seemed cost effective though.

criley2 14d ago

>There are numerous benchmarks that measure cost per task, which factors out tokens entirely. Gemini 3.8 flash is significantly lower than Sol on basically all of them https://artificialanalysis.ai/#cost-tabs https://artificialanalysis.ai/#cost-tabs Not sure if you read your own link but Sol 56 high ranks smack between Gemini 3.8 flash medium and high. Gemini 3.8 flash comes in as more expensive per task than Sol 56 high according to artificial analysis. Luna high is literally 30X cheaper than Gemini 3.8 flash high. You can limit the model viewer and they're getting better at testing multiple effort levels now: https://artificialanalysis.ai/?models=gpt-5-6-sol-medium%2Cgemini-3-8-flash-medium%2Cgemini-3-8-flash%2Cgpt-5-6-sol-high%2Cgpt-5-6-luna-high%2Cgpt-5-6-sol-xhigh#cost-tabs https://artificialanalysis.ai/?models=gpt-5-6-sol-medium%2Cg... One reason is clear: Sol uses dramatically fewer output tokens than Gemini 38 flash https://artificialanalysis.ai/?models=gemini-3-8-flash%2Cgemini-3-8-flash-medium%2Cgpt-5-6-sol-high%2Cgpt-5-6-sol-medium%2Cgpt-5-6-luna-high&cost=intelligence-vs-cost-per-task#output-tokens-tabs https://artificialanalysis.ai/?models=gemini-3-8-flash%2Cgem...

PunchTornado 14d ago
I open the link and I see Flash 3.8 high at 0.58 and Sol at 0.95. I don't understand why you say that "Sol 56 high ranks smack between Gemini 3.8 flash medium and high" but that is clearly wrong.
- criley2 14d ago
  On cost per intelligence task, Gemini38flash and Sol56 trade back and forth on cost depending on effort level. https://i.imgur.com/zPaWPXx.png https://i.imgur.com/zPaWPXx.png As seen in this image, literally: Sol56 high ranks in between Gemini 38 medium and high. The image proves it. I also included Sol56 xhigh, which ranks above even Gemini38 high.

refactor_master 14d ago

I don't know if my code is just "complex", but I find that Luna on max ignores the surrounding style and completely ignores logical consequences of a change, like just writing `del arg1, del arg2, ...` instead of dropping it from the surrounding code. All LLMs make questionable decisions at times, but Luna requires so much guidance that it's faster to just type it out yourself. What kind of routine tasks can one accomplish with such a model?

kasey_junk 14d ago
Do you have code formatters, linters and static analysis? I can get extremely dumb models to get our code style correct because of those guard rails and a specific style document.