5 ms·
I don't think SWE Verified is an ideal benchmark, as the solutions are in the training dataset.
by fuddle 9mo ago
I don't think SWE Verified is an ideal benchmark, as the solutions are in the training dataset.
- joshuahedlund 9mo agoI would love for SWE Verified to put out a set of fresh but comparable problems and see how the top performing models do, to test against overfitting.