6 ms·
I'm curious how they evaluated model quality. The only information I could find is "Quality: Index based on several quality benchmarks".
by idiliv 3y ago
I'm curious how they evaluated model quality. The only information I could find is "Quality: Index based on several quality benchmarks".
- Gcam 3y agoQuality index is equally-weighted normalized values of Chatbot Arena Elo Score, MMLU, and MT Bench. We have a bit more information in the FAQ: https://artificialanalysis.ai/faq https://artificialanalysis.ai/faq but thanks for the feedback, will look into expanding more on how the normalization works. We are thinking of ways to improve this generalized metric. A sticking point is quality can of course be thought of from different perspectives, reasoning, knowledge (retrieval), use-case specific (coding, math, readability), etc. This is why show individual scores on home page and models page: https://artificialanalysis.ai/models https://artificialanalysis.ai/models