6 ms·
It's easy to game and human evaluation data has its trade-offs, but it's way easier to fake public benchmark results. I wish we had a source of high quality pri
by nabakin 6mo ago
It's easy to game and human evaluation data has its trade-offs, but it's way easier to fake public benchmark results. I wish we had a source of high quality private benchmark results across a vast number of models like Lmarena. Having high quality human evaluation data would be a plus too.
- moffkalast 6mo agoWell there was this one [0] which is a black box but hasn't really been kept up to date with newer releases. Arguably we'd need lots of these since each one could be biased towards some use case or sell its test set to someone with more VC money than sense. [0] https://oobabooga.github.io/benchmark.html https://oobabooga.github.io/benchmark.html
- nabakin 6mo agoI know Arc AGI 2 has a private test set and they have a good amount of results[0] but it's not a conventional benchmark. Looking around, SWE Rebench seems to have decent protection against training data leaks[1]. Kagi has one that is fully private[2]. One on HuggingFace that claims to be fully private[3]. SimpleBench[4]. HLE has a private test set apparently[5]. LiveBench[6]. Scale has some private benchmarks but not a lot of models tested[7]. vals.ai[8]. FrontierMath[9]. Terminal Bench Pro[10]. AA-Omniscience[11]. So I guess we do have some decent private benchmarks out there. [0] https://arcprize.org/leaderboard https://arcprize.org/leaderboard [1] https://swe-rebench.com/about https://swe-rebench.com/about [2] https://help.kagi.com/kagi/ai/llm-benchmark.html https://help.kagi.com/kagi/ai/llm-benchmark.html [3] https://huggingface.co/spaces/DontPlanToEnd/UGI-Leaderboard https://huggingface.co/spaces/DontPlanToEnd/UGI-Leaderboard [4] https://simple-bench.com/ https://simple-bench.com/ [5] https://agi.safe.ai/ https://agi.safe.ai/ [6] https://livebench.ai/ https://livebench.ai/ [7] https://labs.scale.com/leaderboard https://labs.scale.com/leaderboard [8] https://www.vals.ai/about https://www.vals.ai/about [9] https://epoch.ai/frontiermath/ https://epoch.ai/frontiermath/ [10] https://github.com/alibaba/terminal-bench-pro https://github.com/alibaba/terminal-bench-pro [11] https://artificialanalysis.ai/articles/aa-omniscience-knowledge-hallucination-benchmark https://artificialanalysis.ai/articles/aa-omniscience-knowle...