5 ms·
what the judge disagreement rate looked like between runs?
by abdulhafeth 8d ago
what the judge disagreement rate looked like between runs?
- tazo-lezhava 6d agoThanks for the question! Here is an easy tool to compare all the different metrics across LLM judges - https://quissly.com/benchmarks/search/all https://quissly.com/benchmarks/search/all, but on a high level: Claude Sonnet 5 vs published: 82.2%, kappa 0.745; Claude vs Gemini 2.5: 81.8%, kappa 0.738. All the details are in the paper as well, of course.