6 ms·
SWE-Bench pro is pretty much useless now even though many ppl still look at it. OpenAI published a report yesterday saying so as well. Only look at DeepSWE and
by osti 2mo ago
SWE-Bench pro is pretty much useless now even though many ppl still look at it. OpenAI published a report yesterday saying so as well. Only look at DeepSWE and FrontierCode right now for coding imo.
- SirMaster 2mo agoAmazing, a company that does poorly in a benchmark says that benchmark is useless...
- osti 2mo agoSWE-bench series just aren't that great by today's standard, even Anthropic previously stated Claude had memorized solutions for the non Pro version of the benchmark, I suspect the recent increase in the score for the Pro version probably also had similar behaviors. But anyway, I think it's pretty useless to look at SWE Bench's now when other way better benchmarks exist.
- tripleee 2mo ago> Anthropic previously stated Claude had memorized solutions for the non Pro version of the benchmark yeah that was the point of introducing the Pro version