lieret

6 ms·

1.
by lieret 5mo ago
We disallow decompilation because we want this task to answer the question of "how well can LMs program from scratch if there's no structure given", not "how well can LMs stitch together decompiled pieces".
2.
Show HN: New Benchmark from SWE-bench team is 0% solved(programbench.com)
24 points by lieret 5mo ago|3 comments
3.
Show HN: All the LM solutions on SWE-bench are bloated compared to humans(twitter.com)
1 points by lieret 7mo ago|0 comments
4.
Show HN: New eval from SWE-bench team evalutes LMs based on goals not tickets(codeclash.ai)
5 points by lieret 11mo ago|1 comments
5.
by lieret 1y ago
[On swe-bench team] We read and analyzed a lot of trajectories but seems like only recently models have started to exploit this in a small fraction of instances. But yes, clearly shouldn't have happened (and is now fixed in the new con
6.
by lieret 1y ago
[On the SWE-bench team] As someone pointed out SWE-bench Verified is a subset of tasks that were reviewed to be solvable (i.e., have enough context in the task description) as well are scored with unit tests that aren't overly specifi
7.
by lieret 1y ago
[Also on the SWE-bench team] Part of the reason why this didn't surface earlier was that it only seems to affect more recent models, maybe the result of reward hacking during posttraining. We're currently working on making traject
8.
Show HN: Randomly switching between LMs at every step boosts SWE-bench score(swebench.com)
5 points by lieret 1y ago|1 comments
9.
by lieret 1y ago
I think gpt-5-mini should really help them. At least from these benchmark scores, there probably shouldn't be a huge performance degradation for letting gpt-5-mini drive most of the workflow. Of course users might still want to just ru
10.
GPT-5 on SWE-bench: Cost and performance deep-dive(mini-swe-agent.com)
4 points by lieret 1y ago|3 comments
11.
by lieret 1y ago
We evaluated the new GPT models with a minimal agent on SWE-bench verified. GPT-5 scores 65%, mini 60%, nano 35%. Still behind Opus 5 (68%), on par with Sonnet 4 (65%). But a lot cheaper, especially mini! Cost is tricky to compare with agen
12.
by lieret 1y ago
Sorry, I missed that! That's a little bit out of the scope of this project (because we were aiming for the bare minimum of what is needed to get a performative agent — and unfortunately learning from mistake also isn't measured by
13.
Show HN: New SWE-bench leaderboard compares LMs without fancy agent scaffolds(swebench.com)
2 points by lieret 1y ago|0 comments
14.
Show HN: Mini-swe-agent achieves 65% on SWE-bench in 100 lines of python(github.com)
7 points by lieret 1y ago|4 comments
15.
by lieret 1y ago
In 2024, we developed SWE-bench and SWE-agent at Princeton University and helped kickstart the coding agent revolution. Back then, LMs were optimized to be great at chatting, but not much else. This meant that agent scaffolds had to get ver