5 ms·
Has anyone bounced these kinds of Agents.md through practical benchmarks?
by acgourley 2mo ago
Has anyone bounced these kinds of Agents.md through practical benchmarks?
- jumploops 2mo agoI’ve thrown my agentic workflow at Terminal Bench 2.1 and it found a bunch of issues (aka failed tests) because the prompts are “bad” and verifiers are overly specific. As an example, there’s a task that asks to make a MIPs interpreter to run Doom, and save a frame at something like /tmp/frame.bmp My spec-driven flow was like “this is useless, let’s record frames like /tmp/frame-N.bmp” Instant fail.
- what 2mo agoHuh? The task was to write a frame to a specific file, your workflow failed.
- jumploops 2mo agoReminder, this is in the context of "dumb human" prompting. The task is to build a MIPS interpreter to run Doom. The "failed" workflow decided that it couldn't prove Doom was booting correctly by just checking one frame, and it decided to check multiple frames (hence /tmp/frame-N.bmp). Arguably a better solution, but obviously fails a brittle test case. The MIPS interpreter worked, but the verifier doesn't actually check that it works, just that a specific frame is logged.