6 ms·
Ignoring the obvious ai slop webpage. I don't really trust the benchmark. It seems either pretty saturated, or inconsistent just based on the results.
by gunalx 7d ago
Ignoring the obvious ai slop webpage. I don't really trust the benchmark. It seems either pretty saturated, or inconsistent just based on the results.
- XCSme 7d agoWhat seems inconsistent? The coverage is quite small, only 22 tests. It's more to compare the cost/speed/consistency between models, given the same tasks.
- gunalx 6d agoRight. I got the feeling of it being saturated because all the top 5 fully completed it.
- XCSme 6d agoYeah, it's hard to find a single simple task that all models fail on, in low context length conditions. Also because models now are actually not that good on knowing things (domain knowledge), as they rely more on web search on tool use. So if I added a question, about some obscure fact, probably the SOTA models would fail it, but in practice they would find it with web search enabled. Not sure how to handle that. This is also why Gemini is on top, it's good enough at coding and instructions following, while having by far best general and domain specific knowledge.