Edward40

7 ms·

1. 1.▲
   by Edward40 7d ago

   Claude Code is an opinionated harness. I’d love to see whether neutral harnesses can challenge the monolithic harness narrative.
2. 2.▲
   by Edward40 14d ago

   We'd rather not mess with system prompts, we just eval them as shipped to keep the results reproducible.
3. 3.▲
   by Edward40 14d ago

   I also prefer using vanilla Pi over Oh My Pi.
4. 4.▲
   by Edward40 14d ago

   Thanks! V1.1 is set to add more harnesses and evaluate a broader range of models. We're moving to a full harness × model matrix to uncover interaction effects and release a complete harness-model compatibility map.
5. 5.▲
   by Edward40 14d ago

   Good points! That’s a known limitation of our v1.0 benchmark with Claude Code. For v1.1, we're expanding both harnesses and models to evaluate the full harness × model matrix. This will highlight interaction effects (showing which harn
6. 6.▲
   by Edward40 14d ago

   We initially tried using the mean (average), but a few extreme outliers caused Claude Code's cost to look far higher than it typically is.
7. 7.▲
   by Edward40 14d ago

   Version 1.1 will add more harnesses and evaluate a broader set of models. Rather than holding the model fixed, we plan to evaluate the full harness × model matrix to reveal interaction effects (which harnesses work best with which models) a
8. 8.▲
   by Edward40 14d ago

   The Exo harness is the most interesting one in the benchmark, as it has the potential to complete tasks with fewer turns and cost.