7 ms·

I am trying to estimate if my reaction to seeing GPT-5.6 Sol last on that list is reasonable or or mostly emotional and find that I have no way of telling.

CompoundEyes 4d ago
I do think it’s the wizard not the wand at this point given a decent model. These benchmarks don’t have the wizard. Otherwise I wouldn’t see others in the exact same codebase struggle and underutilize agents while others thrive using the exact same ones.
- howunfortunate 4d ago
  In other words, we're still in the era of centaur chess.
didgeoridoo 4d ago
Sol failing mostly on “unverified assumptions” and rarely hitting “integration errors” seems about right to me. I think Sol is second only to Astra (and miles ahead of even Fable) in architecting & engineering the right implementation — but only if you are extremely specific and provide tight guidelines and guardrails. If you give it a one-liner… you’re going to have a bad (SHA-256-hash-verified) time.
- enraged_camel 4d ago
  >> I think Sol is second only to Astra (and miles ahead of even Fable) in architecting & engineering the right implementation — but only if you are extremely specific and provide tight guidelines and guardrails. To me, having to give extremely specific instructions and provide tight guidelines and guardrails defeats the purpose of agentic coding agents almost completely. At that point I might as well do the task myself. With Fable I can start with a general ask like "I'm trying to do X, can you investigate and tell me what the shape would look like" and have it poke around and think, ask me questions with single-choice or multiple-choice answers, then break the task into small chunks, each of which becomes a ticket. With Astra, it's like pulling teeth. It often does not understand what I'm trying to do, takes things literally, does not go above and beyond (i.e. infer intent), and stops way too short of the actual goal. I have to constantly prod it and it's frankly exhausting.
  - zeven7 4d ago
    I agree somewhat with the way the agents behave but feel the opposite reaction. With Fable, I get exhausted because it's always dumping out paragraphs of text that explain one approach but have some secret gotcha thrown out in the last two sentences. Then I have to pause and consider the caveat and if it matters and it happens every single time Fable responds and that constantly needing to make a decision that could radically change the approach gives me decision fatigue. I much prefer how much more decisive Astra can be.
    - enraged_camel 4d ago
      I haven't run into the verbosity issue since they added the "Concise" outputStyle, and Fable 5.1 has been even better about not outputting word slops.
    - glub 4d ago
      I'm the opposite. Every time I've let Sol/Astra be decisive, I ended up with an overengineered mess. I much prefer getting alerted when there's more than 1 approach to the problem and it's discovered mid-implementation. I don't want to do the grunt work of writing code, but I do want to know the architecture and be responsible for the decisions. Fable is also very good at pushing back when I propose something that will cost me. E.g. I'm working on a configuration layer above nix to manage my homelab fleet declaratively, and I tend to get into "config as new language", where Fable just goes - let's not do that.
      - zeven7 3d ago
        > Fable is also very good at pushing back Oh man! This also is a pet peeve of mine with Fable. I will look at what it's doing and say "Shouldn't it be done this way?" and then it will spend forever arguing with me that it should be done the way it wanted to do it. It seems to get stuck in a certain way of thinking and will insist its way is right until I can really prove it - or just go over to Astra.
- guybedo 4d ago
  SHA-256-hash-verified sealed package artifact with automatic reconciliation system p95<0.5ms
- janaksunil 4d ago
  makes sense
WD-42 4d ago
Why would you get emotional over a model? They got you that good?
CuriouslyC 4d ago
Any bench that puts GLM 5.3 ahead of 5.6 Sol is highly sus. They've been my two daily drivers since release, and I like GLM 5.3, but it's definitely not better than Sol, it's more ~Terra, while being significantly slower.
- switchbak 4d ago
  I had exactly the same thoughts. I often have similar thoughts on other benchmark sites, where supposed performance is way off base from my experience. I’m not sure what the methodology of these are, but they certainly don’t match what I experience. Maybe I need to look deeper for relevant benchmarks.
- janaksunil 4d ago
  the tasks on the benchmar are long horizon swe tasks - where GLM does surprisingly well
beefsack 4d ago
There's an issue with GPT-5.6 Sol where it sometimes starts mixing thinking with output and stops working[1]. Once it starts doing that, the session is essentially cooked and you need to do a bit of gymnastics if you want to recover it. This happens to me more commonly in large projects (>100k LOC) and in those projects it seems to happen every few sessions. I feel this specific benchmark would be impacted by this more than the smaller contrived benchmarks. [1]: https://github.com/openai/codex/issues/37524 https://github.com/openai/codex/issues/37524
- ImL1s 4d ago
  [flagged]
dimgl 4d ago
I found 5.6 Sol to be extremely underwhelming.
jeffybefffy519 4d ago
Honestly matches my experience with Sol and the fact that all the random message boards appearing on the internet are attributed to Sol then it makes sense that its the one which makes up most requirements.

CompoundEyes 4d ago

I do think it’s the wizard not the wand at this point given a decent model. These benchmarks don’t have the wizard. Otherwise I wouldn’t see others in the exact same codebase struggle and underutilize agents while others thrive using the exact same ones.

howunfortunate 4d ago
In other words, we're still in the era of centaur chess.

didgeoridoo 4d ago

Sol failing mostly on “unverified assumptions” and rarely hitting “integration errors” seems about right to me. I think Sol is second only to Astra (and miles ahead of even Fable) in architecting & engineering the right implementation — but only if you are extremely specific and provide tight guidelines and guardrails. If you give it a one-liner… you’re going to have a bad (SHA-256-hash-verified) time.

enraged_camel 4d ago
>> I think Sol is second only to Astra (and miles ahead of even Fable) in architecting & engineering the right implementation — but only if you are extremely specific and provide tight guidelines and guardrails. To me, having to give extremely specific instructions and provide tight guidelines and guardrails defeats the purpose of agentic coding agents almost completely. At that point I might as well do the task myself. With Fable I can start with a general ask like "I'm trying to do X, can you investigate and tell me what the shape would look like" and have it poke around and think, ask me questions with single-choice or multiple-choice answers, then break the task into small chunks, each of which becomes a ticket. With Astra, it's like pulling teeth. It often does not understand what I'm trying to do, takes things literally, does not go above and beyond (i.e. infer intent), and stops way too short of the actual goal. I have to constantly prod it and it's frankly exhausting.
- zeven7 4d ago
  I agree somewhat with the way the agents behave but feel the opposite reaction. With Fable, I get exhausted because it's always dumping out paragraphs of text that explain one approach but have some secret gotcha thrown out in the last two sentences. Then I have to pause and consider the caveat and if it matters and it happens every single time Fable responds and that constantly needing to make a decision that could radically change the approach gives me decision fatigue. I much prefer how much more decisive Astra can be.
  - enraged_camel 4d ago
    I haven't run into the verbosity issue since they added the "Concise" outputStyle, and Fable 5.1 has been even better about not outputting word slops.
  - glub 4d ago
    I'm the opposite. Every time I've let Sol/Astra be decisive, I ended up with an overengineered mess. I much prefer getting alerted when there's more than 1 approach to the problem and it's discovered mid-implementation. I don't want to do the grunt work of writing code, but I do want to know the architecture and be responsible for the decisions. Fable is also very good at pushing back when I propose something that will cost me. E.g. I'm working on a configuration layer above nix to manage my homelab fleet declaratively, and I tend to get into "config as new language", where Fable just goes - let's not do that.
    - zeven7 3d ago
      > Fable is also very good at pushing back Oh man! This also is a pet peeve of mine with Fable. I will look at what it's doing and say "Shouldn't it be done this way?" and then it will spend forever arguing with me that it should be done the way it wanted to do it. It seems to get stuck in a certain way of thinking and will insist its way is right until I can really prove it - or just go over to Astra.
guybedo 4d ago
SHA-256-hash-verified sealed package artifact with automatic reconciliation system p95<0.5ms
janaksunil 4d ago
makes sense

WD-42 4d ago

Why would you get emotional over a model? They got you that good?

CuriouslyC 4d ago

Any bench that puts GLM 5.3 ahead of 5.6 Sol is highly sus. They've been my two daily drivers since release, and I like GLM 5.3, but it's definitely not better than Sol, it's more ~Terra, while being significantly slower.

switchbak 4d ago
I had exactly the same thoughts. I often have similar thoughts on other benchmark sites, where supposed performance is way off base from my experience. I’m not sure what the methodology of these are, but they certainly don’t match what I experience. Maybe I need to look deeper for relevant benchmarks.
janaksunil 4d ago
the tasks on the benchmar are long horizon swe tasks - where GLM does surprisingly well

beefsack 4d ago

There's an issue with GPT-5.6 Sol where it sometimes starts mixing thinking with output and stops working[1]. Once it starts doing that, the session is essentially cooked and you need to do a bit of gymnastics if you want to recover it. This happens to me more commonly in large projects (>100k LOC) and in those projects it seems to happen every few sessions. I feel this specific benchmark would be impacted by this more than the smaller contrived benchmarks. [1]: https://github.com/openai/codex/issues/37524 https://github.com/openai/codex/issues/37524

ImL1s 4d ago
[flagged]

dimgl 4d ago

I found 5.6 Sol to be extremely underwhelming.

jeffybefffy519 4d ago

Honestly matches my experience with Sol and the fact that all the random message boards appearing on the internet are attributed to Sol then it makes sense that its the one which makes up most requirements.