6 ms·

I'm not sure about all these benchmarks, I did some very simple tests (I have my own benchmarks https://upmaru.com/llm-tests https://upmaru.com/llm-tests) and t

I'm not sure about all these benchmarks, I did some very simple tests (I have my own benchmarks https://upmaru.com/llm-tests https://upmaru.com/llm-tests) and these models fail, not sure if it's the inference provider or the model. They seem to be optimized for benchmarks more than real use cases. Do anything outside their distribution (even if it's not complex) they fail. I Compared Deepseek V4 Flash 0731 (low) to Gemini 3.5 Flash Lite (minimal) and GPT 5.6 Luna (no reasoning) and Deepseek V4 Flash 0731 gets it wrong alot, where as Gemini and 5.6 Luna just gets it done.

theogravity 1mo ago
You're using it on low, that's why. There's a huge difference in performance from low to max effort.
- zacksiri 1mo ago
  I’m comparing same / similar settings between models. I can’t use high on one and low on others it’s not a fair test. Not sure why I was downvoted. But seems the downvoter is quick to downvote anything that doesn’t fit the narrative they’re looking for. I’m just reporting my findings.
  - theogravity 1mo ago
    I did not downvote you. I think it was the way you wrote the comment which made it seem like it couldn't perform in general.
  - kgeist 1mo ago
    I'm not sure it's a fair test either to compare the "low" setting of one model with the "low" setting of another. They're completely different settings that just happen to have the same name.
  - lejalv 1mo ago
    Comparing at similar thinking hasn't much value. You can compare the tiers that have the closest price, that would be more interesting.
    - zacksiri 1mo ago
      Yes, I think a proper comprehensive test would be a better judge of the outcome. I may do round 2 given my first batch of models is already outdated.
  - Palmik 1mo ago
    Low, High and Max, obviously, can't be compared across models. They only mean the model is likely to spend less reasoning effort (~output tokens) with Low than High on the same, *single shot* task. But even in this very post, you can see that Max was actually cheaper than High. If you are using API, you should be comparing based on end-to-end cost or speed or whatever blend of those two matches your cost/time budget.

theogravity 1mo ago

You're using it on low, that's why. There's a huge difference in performance from low to max effort.

zacksiri 1mo ago
I’m comparing same / similar settings between models. I can’t use high on one and low on others it’s not a fair test. Not sure why I was downvoted. But seems the downvoter is quick to downvote anything that doesn’t fit the narrative they’re looking for. I’m just reporting my findings.
- theogravity 1mo ago
  I did not downvote you. I think it was the way you wrote the comment which made it seem like it couldn't perform in general.
- kgeist 1mo ago
  I'm not sure it's a fair test either to compare the "low" setting of one model with the "low" setting of another. They're completely different settings that just happen to have the same name.
- lejalv 1mo ago
  Comparing at similar thinking hasn't much value. You can compare the tiers that have the closest price, that would be more interesting.
  - zacksiri 1mo ago
    Yes, I think a proper comprehensive test would be a better judge of the outcome. I may do round 2 given my first batch of models is already outdated.
- Palmik 1mo ago
  Low, High and Max, obviously, can't be compared across models. They only mean the model is likely to spend less reasoning effort (~output tokens) with Low than High on the same, *single shot* task. But even in this very post, you can see that Max was actually cheaper than High. If you are using API, you should be comparing based on end-to-end cost or speed or whatever blend of those two matches your cost/time budget.