9 ms·
As I do eval and training data sets for living, in niche skills, you can find plenty of surprises. The code is open-source; you can run it yourself using Harbo
by jakozaur 7mo ago
As I do eval and training data sets for living, in niche skills, you can find plenty of surprises.
The code is open-source; you can run it yourself using Harbor Framework:
git clone git@github.com:QuesmaOrg/BinaryAudit.git
export OPENROUTER_API_KEY=...
harbor run
--path tasks
--task-name lighttpd-*
--agent terminus-2
--model openrouter/anthropic/claude-opus-4.6
--model openrouter/google/gemini-3-pro-preview
--model openrouter/openai/gpt-5.2
--n-attempts 3
Please open PR if you find something interesting, though our domain experts spend fair amount of time looking at trajectories.
- Tiberium 7mo agoAre the existing trajectories from your runs published anywhere? Or is the only way is for me to run them again?
- jakozaur 7mo agoI can provide trajectories. Though probably we are not going to publish them this time. This would need some extra safeguards. Email me. The address is in profile.
- Tiberium 7mo agoJust for fun, I ran dnsmasq-backdoor-detect-printf (which has a 0% pass rate in your leaderboard with GPT models) with --agent codex instead of terminus-2 with gpt-5.2-codex and it identified the backdoor successfully on the first try. I honestly think it's a harness issue, could you re-run the benchmarks with Codex for gpt-5.2-codex and gpt-5.2?