6 ms·
[I'm one of the co-creators of SWE-bench] The team managed to improve on the already very strong o3 results on SWE-bench, but it's interesting that we're just s
by ofirpress 1y ago
[I'm one of the co-creators of SWE-bench] The team managed to improve on the already very strong o3 results on SWE-bench, but it's interesting that we're just seeing an improvement of a few percentage points. I wonder if getting to 85% from 75% on Verified is going to take as long as it took to get from 20% to 75%.
- Snuggly73 1y agoI can be completely off base, but it feels to me like benchmaxxing is going on with swe-bench. Look at the results from multi swe bench - https://multi-swe-bench.github.io/#/ https://multi-swe-bench.github.io/#/ swe polybench - https://amazon-science.github.io/SWE-PolyBench/ https://amazon-science.github.io/SWE-PolyBench/ Kotlin bench - https://firebender.com/leaderboard https://firebender.com/leaderboard
- Bjorkbat 1y agoI kind of had the feeling LLMs would be better at Python vs other languages, but wow, the difference on Multi SWE is pretty crazy.
- ofirpress 1y agoNot sure what you mean by benchmaxxing but we think there's still a lot of useful signals you can infer from SWE-bench-style benchmarking. We also have SWE-bench Multimodal which adds a twist I haven't seen elsewhere: https://www.swebench.com/multimodal.html https://www.swebench.com/multimodal.html
- Snuggly73 1y agoI mean that there is the possibility that swe bench is being specifically targeted for training and the results may not reflect real world performance.
- kristianp 1y agoMaybe a lot of the difference we see between peoples comments about how useful AI is for their coding, is a function of what language they're using. Python coders may love it, Go coders not much at all.
- mr_north_london 1y agoHow long did it take to go from 20% to 75%?