6 ms·

They do disclose that they scored much lower than Fable on SWEBench Pro, which is a pretty high-quality benchmark. I think it's partially just about what they c

by thurn 2mo ago

They do disclose that they scored much lower than Fable on SWEBench Pro, which is a pretty high-quality benchmark. I think it's partially just about what they choose to emphasize...

bel8 2mo ago
SWEBench Pro should be ignored until they fix it or disprove the broken task accusations.
mchinen 2mo ago
I totally missed that, because in the charts they showcase for coding, the SWEBench score is not present, they only include it at the end of the post in tables. Hmm. Great catch.
- saberience 2mo ago
  The SWEBench benchmarks are really gamed at this point and should not be trusted period. The solutions are effectively in the training sets and have been for a while.
  - mnicky 2mo ago
    SWE Bench Pro is completely different benchmark than SWE Bench (e.g. Verified) suite was. It only copied the name.
    - saberience 2mo ago
      SWE Bench Pro is also gamed and shouldn't be trusted.
      - mnicky 2mo ago
        Then we are left with what? FrontierCode maybe? IIRC that one evaluates not only if tests pass but also code quality - e.g. whether the maintainer would accept the pull request as is.
      - Sabinus 2mo ago
        Why that benchmark in particular?
simianwords 2mo ago
> SWEBench Pro, which is a pretty high-quality benchmark No, doesn't seem like it https://openai.com/index/separating-signal-from-noise-coding-evaluations/ https://openai.com/index/separating-signal-from-noise-coding...
danielsamuels 2mo ago
It's worth noting that OpenAI recently came out saying, "We don't think SWEBench Pro is worth reporting any more" - https://openai.com/index/separating-signal-from-noise-coding-evaluations/ https://openai.com/index/separating-signal-from-noise-coding...
- 37374848 2mo ago
  [flagged]
  - William_BB 2mo ago
    People will downvote you because this comment is "not appropriate" for HN, but there were countless conversations on HN about how important these benchmarks are. I am literally LOLing at HN right now
    - Sabinus 2mo ago
      You're free to lol but at least put a substantive comment with it. 'Lol' by itself is low quality.
    - anthonyrstevens 2mo ago
      It's almost like there are multiple different people commenting on HN.
lwansbrough 2mo ago
Didn't they also just post about how SWEBench is broken?
futureshock 2mo ago
There has been a lot of chatter ever since the Mythos scores had been release that SWEbench pro had major contamination and that Mythos had memorized many questions that lacked the context to be solvable on their own. And now with OpenAI saying a large number of the questions are broken, I think it's worth taking that single outlier benchmark with some salt when the overall trend is that 5.6 is very competitive with Mythos at about half the price.

by thurn 2mo ago

They do disclose that they scored much lower than Fable on SWEBench Pro, which is a pretty high-quality benchmark. I think it's partially just about what they choose to emphasize...

bel8 2mo ago

SWEBench Pro should be ignored until they fix it or disprove the broken task accusations.

mchinen 2mo ago

I totally missed that, because in the charts they showcase for coding, the SWEBench score is not present, they only include it at the end of the post in tables. Hmm. Great catch.

saberience 2mo ago
The SWEBench benchmarks are really gamed at this point and should not be trusted period. The solutions are effectively in the training sets and have been for a while.
- mnicky 2mo ago
  SWE Bench Pro is completely different benchmark than SWE Bench (e.g. Verified) suite was. It only copied the name.
  - saberience 2mo ago
    SWE Bench Pro is also gamed and shouldn't be trusted.
    - mnicky 2mo ago
      Then we are left with what? FrontierCode maybe? IIRC that one evaluates not only if tests pass but also code quality - e.g. whether the maintainer would accept the pull request as is.
    - Sabinus 2mo ago
      Why that benchmark in particular?

simianwords 2mo ago

> SWEBench Pro, which is a pretty high-quality benchmark No, doesn't seem like it https://openai.com/index/separating-signal-from-noise-coding-evaluations/ https://openai.com/index/separating-signal-from-noise-coding...

danielsamuels 2mo ago

It's worth noting that OpenAI recently came out saying, "We don't think SWEBench Pro is worth reporting any more" - https://openai.com/index/separating-signal-from-noise-coding-evaluations/ https://openai.com/index/separating-signal-from-noise-coding...

37374848 2mo ago
[flagged]
- William_BB 2mo ago
  People will downvote you because this comment is "not appropriate" for HN, but there were countless conversations on HN about how important these benchmarks are. I am literally LOLing at HN right now
  - Sabinus 2mo ago
    You're free to lol but at least put a substantive comment with it. 'Lol' by itself is low quality.
  - anthonyrstevens 2mo ago
    It's almost like there are multiple different people commenting on HN.

lwansbrough 2mo ago

Didn't they also just post about how SWEBench is broken?

futureshock 2mo ago

There has been a lot of chatter ever since the Mythos scores had been release that SWEbench pro had major contamination and that Mythos had memorized many questions that lacked the context to be solvable on their own. And now with OpenAI saying a large number of the questions are broken, I think it's worth taking that single outlier benchmark with some salt when the overall trend is that 5.6 is very competitive with Mythos at about half the price.