5 ms·
Are there any objective metrics/ benchmarks that people test harnesses by? There are just so many now that it's hard to personally test them all or just trust
by yewenjie 23d ago
Are there any objective metrics/ benchmarks that people test harnesses by?
There are just so many now that it's hard to personally test them all or just trust the vibes.
- andyk 23d agoandy here (headlong post author). terminal bench 3 is pretty popular for comparing different harnesses using the same underlying model (it's another laude project actually). artificial analysis has an index. you can look at the model cards of popular model releases- they tend to have the most popular current benchmarks on them. w/ headlong we decided to announce it before we've benchmarked it. we mostly wanted to informally share our experiences w/ it in this initial post. we plan to do some benchmarking coming up here soon tho
- embedding-shape 23d agoDon't use any public benchmarks, every single one is worthless for your own use cases essentially. Spend a day or two going through your existing chat sessions, and create your own private benchmark with test cases based on real tasks, that you don't share with anyone nor publicly. Make it easy to add/remove new harnesses and model combinations, make it give you a final score, ideally avoid using other LLMs for scoring, then use this to figure out if the new model/harness actually improves things for you. I've been doing this for some time, and while most new releases show big increases in the benchmarks/evaluations, my own benchmark usually barely moves.
- krashidov 23d ago> make it give you a final score what does this mean exactly? A scored based on what?
- embedding-shape 23d agoFor translations, the score is basically 1 or 0. For some tasks, the least amount of LOC gives the highest score, and so on. Basically, you need to figure out how to score it, so you can compare scores across agents/models.