5 ms·
To what degree is this a harness/system prompt problem? Models maybe should implement new stuff with as little impact on the existing stuff as possible by defau
by ajwin 2mo ago
To what degree is this a harness/system prompt problem? Models maybe should implement new stuff with as little impact on the existing stuff as possible by default? A simple system prompt for it to always check the code after task completion for proper simplifications, abstractions and cleanups before returning to the user? Instructions to retain "story like" readability of the code.
- dhorthy 2mo agoI agree this is an option, and the next thing on my radar is to try with a more realistic "factory-shaped" harness where you have feedback from linters and other models after each coding episode that refines the architecture. For readability specifically, I've found it hard to get the models to do this with prompting. If you've talked to opus/fable for a long time on prose writing you probably felt this too
- igregoryca 2mo agoAt least for Claude Code, putting "run /simplify at the end" in an "implement the plan" skill helps a little. It still often leaves new code in bizarre places, and/or with bad/alien-sounding names and comments.
- dhorthy 2mo agoYeah I would hold that models don’t know how to simplify because most rl/benchmarks doesn’t penalize complexity
- deleted 2mo ago[deleted]
- lostdog 2mo agoClearly the first step of slopbench2 should be to have the agent first write its own harness!
- renezander030 2mo ago[flagged]