5 ms·
I built exactly this using my own codebases. The setup isn’t that complex; split the git history to just before the change, sandbox the agent with everything th
by springtimesun 4d ago
I built exactly this using my own codebases. The setup isn’t that complex; split the git history to just before the change, sandbox the agent with everything they will need at that commit and lightly modify rules so they don’t go searching outside the box. Then they get the same prompt (usually the ticket that began the work) and are graded against the accepted PR.
The thing that takes the most time is finding the examples. In my real dev flow it’s rarely ticket -> PR -> merge, things bounce around a lot more. So, even though the stated goal is to get away from one shots, that is basically the environment you have to set or else test for specific other outcomes (e.g. agent stopped and raised a question when it realized x).
It takes time to do, but I would really recommend it. Now I can push new open models through the batteries and see how they line up to past ones in a few days (I run them locally, it’s slow). It moves my sense of x model is good at y and bad at z to from vibes to a better heuristic (these still run at temp 1, heuristic is the correct way to think about outcomes IMO). It grounds it in your actual code and problem space.
My takeaway from my testing: in Rails or front end codebases, most models I test are competent and with a human in the loop they would accomplish their goal of getting to a mergeable PR. They are not as good as Claude and since I pay subsidized rates via subscription Claude still gets first pass. They are very worthwhile to layer in as reviewers and catch many issues. My anxiety about a rug pull by the frontiers has been turned way down. I would have to adapt to a local only flow, but it wouldn’t be much adaptation and the opens can deliver in their current state.
- janaksunil 4d agohey this seems really interesting - what prompted you to test multiple agents on your codebase?
- springtimesun 4d agoCuriosity and anxiety. I rebuilt my entire workflow around agents so the unease that the frontiers would change something (access, pricing, availability) and lock me out of that were high. Also why I spent way too much on hardware (at least that can be deducted). Now the whole stack could run in my house and I feel much better about the situation. Once I got the testing going though it is worth it for its own pursuit. Building processes around the dev process and trying to get the best outcomes is at least as fun to me as actually delivering client code. For the first time in my tech career I feel like I’m in a place with no maps. No one has done my experiments yet. I have a custom quant of K3 at Q5 that lets me get 10 tok/s on a CPU inference box (admittedly you need a 72GB Blackwell also). As far as I can tell no one else has done this. It’s such an exciting time!
- janaksunil 4d agodo you have some time to chat? janak@withspecific.com
- springtimesun 3d ago[dead]
- ignoramous 3d agoI have a similar calibration workflow, too [0]; but I am under no illusion that my personal benchmarks remain secret. Once we send code & prompts to the providers, it can no longer be considered private. [0] And came to the same conclusion as you did: Claudes were better but not by much: https://news.ycombinator.com/item?id=48654635 https://news.ycombinator.com/item?id=48654635
- mahogany 3d agoDon't you need to run each model several times and take some average since their outputs can change for a fixed prompt? How do you test for that?
- springtimesun 3d ago[dead]
- yyhhsj0521 3d agoHow do you grade them against the accepted PR?