5 ms·

Artificial Analysis Intelligence Index v4.2

158 points by nojs 12d ago

6thbit 12d ago
Why have they not included ARC-Agi-3 on their index? Clearly that would move things around.
- thereitgoes456 12d ago
  Because playing a video game isn’t relevant to which AI people might want to use.
  - dannyw 11d ago
    "Video game" is the medium, the challenge and test is figuring out how to win; when given no instructions, and specifically designed to be private. ARC-AGI takes it very seriously: they've never tested Fable, because they won't run on the eval set without ZDR. Another possible way to look at this is that any benchmark reporting Fable scores is potentially contaminated.
lousken 12d ago
How to view the previous version to compare?
- redox99 12d ago
  Images from their X account
nthypes 12d ago
What version the intelligence vs cost graph is using? they didn't ran v4.2 to all models.
- AnodicElegy 12d ago
  It's very different than it was earlier today, so pretty sure it's v4.2.
redox99 12d ago
They realized Astra having the same score as Sol was silly so they rushed to update the index so it fits what people expect. The old index was clearly bad (Astra is way better than Sol) but it's also unscientific to tweak it like this.
- paimapi 12d ago
  is any of this 'scientific'? does AA allow peer review of its processes? are these published in journals of at least medium impact? what are the sample sizes? how grounded is their mechanistic reasoning? like they have words that are dressed in scientific language on their site like "We estimate a 95% confidence interval for Artificial Analysis Intelligence Index of less than ±1% - based on experiments with >10 repeats on certain models for all evaluation datasets included in Artificial Analysis Intelligence Index v4.2." but where's the outcome dataset justifying this? how did they get that probability? what was the specific methodology of the tests? what variables did they account for? there's a major difference between scientific sounding and being truly empirically rigorous. the 'research' in AI intelligence feels somehow even less trustworthy than supplement-funded studies because those are at least subjected to scrutiny by peers without profit motives
  - elvin_d 12d ago
    why keeping journals as an argument on the tech site that was always less formal with mandatory institutions but more open source. Journals discredited themselves multiple times. tech is expanding boundaries of scientific methods and AI will push it more.
    - tancop 12d ago
      It's not about journals. If they want to be 100 trustworthy they should release end to end reproducible pipelines for the whole process with everything but the private datasets included, and all design decisions fully documented. And let third party labs audit to confirm that the holdout questions are equal difficulty and similar task types to the public ones.
    - deleted 11d ago
jascha_eng 12d ago
Imo the omniscience index they have has the highest correlation to actual usefulness of the models. https://artificialanalysis.ai/evaluations/omniscience https://artificialanalysis.ai/evaluations/omniscience > measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer. This is so useful because it makes you actually trust a models output. A high score on benchmarks is not as useful because a model overtrained to always answer will give confidently wrong responses. But this index measures how often it is correct while penalizing wrong responses so that a high score means you can trust this model more and when it doesn't know it is more likely to tell you that it really doesn't know rather than making shit up. Fable also performs a lot better than opus 5 here which correlates very strongly with perceived strength despite the models performing similarly on e.g. DeepSWE Astra is a big jump from sol and performs the same or slightly better than fable here.
- Buoylog 12d ago
  This matches what I've seen building anything that uses an LLM for narrow structured output rather than open ended chat, things like classifying a diff into a fixed set of categories or summarizing a change. The aggregate benchmark score barely predicts how it behaves in production. What actually breaks a pipeline is a confident wrong answer on the small slice of inputs that don't fit the pattern it saw during training, not a lack of raw capability. A model that says it isn't sure on the edge cases is far more useful to me than one that scores higher on average but never admits uncertainty, because the wrong but confident output is the one that slips through review unnoticed.
- gizmodo59 12d ago
  I don't agree. The main issue with their scoring/methodology is that the numbers make it seem like 5-6 models have little to no difference when in fact there is a significant difference between fable and opus and sol and astra for example. They are popular mainstream but most of their benchmarks are either not a representation of model strengths enough or they are not doing a good job of showcasing it properly. The fact that muse and 3.8 were high a day back shows they are just the modern version of lmareana for the mass audience and PR stunts.
__jl__ 12d ago
This is really a great achievement: "Astra dominates the output token frontier" Many labs used increased thinking to boost benchmark scores and performance. Most of the Chinese models were doing that for a while. Google and Anthropic as well. Not OpenAI. 5.6 already was much more token efficient than other models and Astra beats Sol in token efficiency by a wide margin. Edit: Just to make the point: Astra (max) has the 2nd highest score and the third lowest output tokens (among the models shown by AA).
- Scaevolus 12d ago
  GPT-6 has a looped transformer / recurrent depth architecture, so it gets some internal CoT reasoning "for free" with no output tokens. This also makes it much harder to monitor its reasoning.
- jsnell 12d ago
  The apparent advantage is exaggerated by them running Astra at six different effort levels, and almost everything else at just the maximum available effort. I don't really understand why they keep doing this. Either run and report everyone at multiple effort levels, or run everyone at only one. But alsi, token efficiency seems pretty artificial? For example tokenizers are different from model to model. The cost/perf Pareto frontier seems a lot more meaningful (and Astra does very well at that too, just to be clear. It seems to be a great model.)
- ssivark 12d ago
  Duh, that's just a benchmarking artifact. If you run one model at 4x recurrence compared to another, you get 4x thinking without increasing the tokens. So of course it's going to dominate the perf at given output token level. The truly sensible comparison is perf -vs- thinking flops (because each model might be a different unknown size, but labs are very secretive about what they're actually running under the hood) or perhaps cost (which can be misleading because of subsidies, but is at least practically relevant in the moment).
  - water-drummer 12d ago
    But the end user doesn't care about flops for closed models. All they care about is how much it ends up costing them.
AnodicElegy 12d ago
This update really gives OpenAI a boost. Not saying there's anything inaccurate or untoward about that, but the timing is unfortunate. It would have looked better had it been done prior to the Fable 5.1 and GPT 6 releases. I guess AA would say that there's no perfect time to do these updates, given the rapid fire pace of releases!
- CuriouslyC 12d ago
  The timing is related to the fact that their benchmark was saying it was the same as Sol, and below Opus 5, when anecdotal reports and other benchmarks strongly disagree. It looked bad for them for their benchmark to disagree with people's lived experience so hard.
  - pixl97 11d ago
    Kind of reminds me when GPU benchmarks used to game the drivers to maximize the FPS. If the benchmark slightly changes the camera view is that cheating? Or is it calling out the cheaters?
theycallmeritik 12d ago
how did you check the prev version to compare?
aurareturn 12d ago
In terms of intelligence per token per cost, OpenAI is really killing it.
sanxiyn 12d ago
It is very unfortunate they upweighted SciCode from 8% to 10%. SciCode is a broken benchmark: see https://arxiv.org/abs/2608.04975 https://arxiv.org/abs/2608.04975.
throwaway13337 12d ago
I have no idea how artificial analysis got to be something anyone took seriously. This is their new benchmark set? A glance at their new index shows that whatever they're measuring, it isn't useful. Spend an hour with gemini 3.8 and tell me that model belongs in 2026. It feels like the model has Alzheimer's. It gets confused about whether what it reads is what it did. Just crazy bad. I haven't tried muse spark 1.3. But it must have been a miracle since 1.2 to hit that rank. Video game journalism vibes all over this.
- nojs 12d ago
  What other benchmarks do you recommend that are more accurate?
  - WASDx 11d ago
    Give a task you have to 3 different models and see what actually works for you. There are no good benchmarks.
- dist-epoch 11d ago
  As the saying goes, Artificial Analysis is the worst benchmarking company, except for all the others.
- Catloafdev 11d ago
  Because it's the best option currently available. It's really easy to shit on AI benchmarks, but that noise is useless unless you're offering a solution or a better benchmark.
jjcm 12d ago
IMO one of the biggest losses of the OpenAI/Cursor breakup will be the loss of OAI models on CursorBench [1]. Their bench has always been one that most-fit my mental model of how good each of these models are. I find AA’s Intelligence index to often be out of alignment with my own subjective evals. [1] https://cursor.com/evals https://cursor.com/evals
stared 11d ago
While I like this index, calling in "Intelligence" might be confusing - it is a mix of coding and knowledge. Compare and contrast with ARC-AGI, BabaIsBench (https://quesma.com/benchmarks/babaisbench/ https://quesma.com/benchmarks/babaisbench/), or MazeBench (https://mazebench.com/blog?post=introducing-mazebench https://mazebench.com/blog?post=introducing-mazebench). In particular, in one Baba Is Bench post (https://quesma.com/blog/baba-is-aug-2026/ https://quesma.com/blog/baba-is-aug-2026/), while quoting a Pareto frontier chart from AA, I noted: > Intelligence Index vs. Cost per Intelligence Index Task from Artificial Analysis. Note that it is based on score of benchmarks like Terminal-Bench v2.1, SciCode, Humanity’s Last Exam, GPQA Diamond - not necessarily fluid intelligence like in abstract puzzle games of ARC-AGI-3 or Baba is You.
__natty__ 11d ago
I wonder if Artificial Analysis could be influenced by certain model companies. Looking at the changelog [0], they updated a few times after new models appeared, so US models progression was much higher than that of other vendors (like when they updated the algorithm after Kimi K3 versus Opus 4.7, so Kimi dropped in the rankings). Or maybe thats just coincidence. [0] https://artificialanalysis.ai/changelog https://artificialanalysis.ai/changelog
- dist-epoch 11d ago
  That would be the fastest way for them to completely torch their company. The only thing they are selling and why people look at them is trust that they do honest evaluations.
  - marmarama 10d ago
    People still buy into "research" from Gartner and Forrester despite it being an open secret for decades that they are pay-to-play. These kinds of "research" companies are there to validate people's preconceived ideas and purchasing decisions rather than being genuinely unbiased. And they are very useful for that, both for consumers and for marketers.
dgacmu 11d ago
I quite wish they'd move to terminal bench 4.0. 2.1 is saturated - there is no world in which Gemini 3.8 Flash is producing better code than Astra or Fable, as the 2.1 results might suggest. The 4.0 results differentiate these models much more effectively. (2.1 is useful for knowing they can all one-shot straightforward scripts, of course.)
- swingboy 11d ago
  DeepSWE has Gemini 3.8 Flash up really high, too.
  - dgacmu 11d ago
    It does and that one also feels kind of saturated for measuring the most advanced models - opus, Gemini, astra, sol, fable, glm, kimi all scoring within statistical noise of each other. (74 +-3% down to 69% +-5% for kimi). It's still providing strong discrimination between weaker models.
ahmedelsama 11d ago
[flagged]
MoreThanMe 11d ago
[flagged]
snezhadianpm 11d ago
[dead]
theagenticleade 11d ago
Are you sure that Fable 5.1 outperforms Astra? Feel like early indications and testing appear to show for the first time a true Agentic experience? This could open the door to a true loop build in every corner of the world.
sheepscreek 11d ago
Seeing the Qwen3.8-27B model among giants is almost comical. It is unparalleled and a true testament to Alibaba’s talent and ingenuity. Can’t overstate how excited I am for newer edge models.
mmmmbbbhb 10d ago
This website lost all trustworthiness with the astra blunder. The benchmarks are worthless.

158 points by nojs 12d ago

6thbit 12d ago

Why have they not included ARC-Agi-3 on their index? Clearly that would move things around.

thereitgoes456 12d ago
Because playing a video game isn’t relevant to which AI people might want to use.
- dannyw 11d ago
  "Video game" is the medium, the challenge and test is figuring out how to win; when given no instructions, and specifically designed to be private. ARC-AGI takes it very seriously: they've never tested Fable, because they won't run on the eval set without ZDR. Another possible way to look at this is that any benchmark reporting Fable scores is potentially contaminated.

lousken 12d ago

How to view the previous version to compare?

redox99 12d ago
Images from their X account

nthypes 12d ago

What version the intelligence vs cost graph is using? they didn't ran v4.2 to all models.

AnodicElegy 12d ago
It's very different than it was earlier today, so pretty sure it's v4.2.

redox99 12d ago

They realized Astra having the same score as Sol was silly so they rushed to update the index so it fits what people expect. The old index was clearly bad (Astra is way better than Sol) but it's also unscientific to tweak it like this.

paimapi 12d ago
is any of this 'scientific'? does AA allow peer review of its processes? are these published in journals of at least medium impact? what are the sample sizes? how grounded is their mechanistic reasoning? like they have words that are dressed in scientific language on their site like "We estimate a 95% confidence interval for Artificial Analysis Intelligence Index of less than ±1% - based on experiments with >10 repeats on certain models for all evaluation datasets included in Artificial Analysis Intelligence Index v4.2." but where's the outcome dataset justifying this? how did they get that probability? what was the specific methodology of the tests? what variables did they account for? there's a major difference between scientific sounding and being truly empirically rigorous. the 'research' in AI intelligence feels somehow even less trustworthy than supplement-funded studies because those are at least subjected to scrutiny by peers without profit motives
- elvin_d 12d ago
  why keeping journals as an argument on the tech site that was always less formal with mandatory institutions but more open source. Journals discredited themselves multiple times. tech is expanding boundaries of scientific methods and AI will push it more.
  - tancop 12d ago
    It's not about journals. If they want to be 100 trustworthy they should release end to end reproducible pipelines for the whole process with everything but the private datasets included, and all design decisions fully documented. And let third party labs audit to confirm that the holdout questions are equal difficulty and similar task types to the public ones.
  - deleted 11d ago

jascha_eng 12d ago

Imo the omniscience index they have has the highest correlation to actual usefulness of the models. https://artificialanalysis.ai/evaluations/omniscience https://artificialanalysis.ai/evaluations/omniscience > measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer. This is so useful because it makes you actually trust a models output. A high score on benchmarks is not as useful because a model overtrained to always answer will give confidently wrong responses. But this index measures how often it is correct while penalizing wrong responses so that a high score means you can trust this model more and when it doesn't know it is more likely to tell you that it really doesn't know rather than making shit up. Fable also performs a lot better than opus 5 here which correlates very strongly with perceived strength despite the models performing similarly on e.g. DeepSWE Astra is a big jump from sol and performs the same or slightly better than fable here.

Buoylog 12d ago
This matches what I've seen building anything that uses an LLM for narrow structured output rather than open ended chat, things like classifying a diff into a fixed set of categories or summarizing a change. The aggregate benchmark score barely predicts how it behaves in production. What actually breaks a pipeline is a confident wrong answer on the small slice of inputs that don't fit the pattern it saw during training, not a lack of raw capability. A model that says it isn't sure on the edge cases is far more useful to me than one that scores higher on average but never admits uncertainty, because the wrong but confident output is the one that slips through review unnoticed.
gizmodo59 12d ago
I don't agree. The main issue with their scoring/methodology is that the numbers make it seem like 5-6 models have little to no difference when in fact there is a significant difference between fable and opus and sol and astra for example. They are popular mainstream but most of their benchmarks are either not a representation of model strengths enough or they are not doing a good job of showcasing it properly. The fact that muse and 3.8 were high a day back shows they are just the modern version of lmareana for the mass audience and PR stunts.

__jl__ 12d ago

This is really a great achievement: "Astra dominates the output token frontier" Many labs used increased thinking to boost benchmark scores and performance. Most of the Chinese models were doing that for a while. Google and Anthropic as well. Not OpenAI. 5.6 already was much more token efficient than other models and Astra beats Sol in token efficiency by a wide margin. Edit: Just to make the point: Astra (max) has the 2nd highest score and the third lowest output tokens (among the models shown by AA).

Scaevolus 12d ago
GPT-6 has a looped transformer / recurrent depth architecture, so it gets some internal CoT reasoning "for free" with no output tokens. This also makes it much harder to monitor its reasoning.
jsnell 12d ago
The apparent advantage is exaggerated by them running Astra at six different effort levels, and almost everything else at just the maximum available effort. I don't really understand why they keep doing this. Either run and report everyone at multiple effort levels, or run everyone at only one. But alsi, token efficiency seems pretty artificial? For example tokenizers are different from model to model. The cost/perf Pareto frontier seems a lot more meaningful (and Astra does very well at that too, just to be clear. It seems to be a great model.)
ssivark 12d ago
Duh, that's just a benchmarking artifact. If you run one model at 4x recurrence compared to another, you get 4x thinking without increasing the tokens. So of course it's going to dominate the perf at given output token level. The truly sensible comparison is perf -vs- thinking flops (because each model might be a different unknown size, but labs are very secretive about what they're actually running under the hood) or perhaps cost (which can be misleading because of subsidies, but is at least practically relevant in the moment).
- water-drummer 12d ago
  But the end user doesn't care about flops for closed models. All they care about is how much it ends up costing them.

AnodicElegy 12d ago

This update really gives OpenAI a boost. Not saying there's anything inaccurate or untoward about that, but the timing is unfortunate. It would have looked better had it been done prior to the Fable 5.1 and GPT 6 releases. I guess AA would say that there's no perfect time to do these updates, given the rapid fire pace of releases!

CuriouslyC 12d ago
The timing is related to the fact that their benchmark was saying it was the same as Sol, and below Opus 5, when anecdotal reports and other benchmarks strongly disagree. It looked bad for them for their benchmark to disagree with people's lived experience so hard.
- pixl97 11d ago
  Kind of reminds me when GPU benchmarks used to game the drivers to maximize the FPS. If the benchmark slightly changes the camera view is that cheating? Or is it calling out the cheaters?

theycallmeritik 12d ago

how did you check the prev version to compare?

aurareturn 12d ago

In terms of intelligence per token per cost, OpenAI is really killing it.

sanxiyn 12d ago

It is very unfortunate they upweighted SciCode from 8% to 10%. SciCode is a broken benchmark: see https://arxiv.org/abs/2608.04975 https://arxiv.org/abs/2608.04975.

throwaway13337 12d ago

I have no idea how artificial analysis got to be something anyone took seriously. This is their new benchmark set? A glance at their new index shows that whatever they're measuring, it isn't useful. Spend an hour with gemini 3.8 and tell me that model belongs in 2026. It feels like the model has Alzheimer's. It gets confused about whether what it reads is what it did. Just crazy bad. I haven't tried muse spark 1.3. But it must have been a miracle since 1.2 to hit that rank. Video game journalism vibes all over this.

nojs 12d ago
What other benchmarks do you recommend that are more accurate?
- WASDx 11d ago
  Give a task you have to 3 different models and see what actually works for you. There are no good benchmarks.
dist-epoch 11d ago
As the saying goes, Artificial Analysis is the worst benchmarking company, except for all the others.
Catloafdev 11d ago
Because it's the best option currently available. It's really easy to shit on AI benchmarks, but that noise is useless unless you're offering a solution or a better benchmark.

jjcm 12d ago

IMO one of the biggest losses of the OpenAI/Cursor breakup will be the loss of OAI models on CursorBench [1]. Their bench has always been one that most-fit my mental model of how good each of these models are. I find AA’s Intelligence index to often be out of alignment with my own subjective evals. [1] https://cursor.com/evals https://cursor.com/evals

stared 11d ago

While I like this index, calling in "Intelligence" might be confusing - it is a mix of coding and knowledge. Compare and contrast with ARC-AGI, BabaIsBench (https://quesma.com/benchmarks/babaisbench/ https://quesma.com/benchmarks/babaisbench/), or MazeBench (https://mazebench.com/blog?post=introducing-mazebench https://mazebench.com/blog?post=introducing-mazebench). In particular, in one Baba Is Bench post (https://quesma.com/blog/baba-is-aug-2026/ https://quesma.com/blog/baba-is-aug-2026/), while quoting a Pareto frontier chart from AA, I noted: > Intelligence Index vs. Cost per Intelligence Index Task from Artificial Analysis. Note that it is based on score of benchmarks like Terminal-Bench v2.1, SciCode, Humanity’s Last Exam, GPQA Diamond - not necessarily fluid intelligence like in abstract puzzle games of ARC-AGI-3 or Baba is You.

__natty__ 11d ago

I wonder if Artificial Analysis could be influenced by certain model companies. Looking at the changelog [0], they updated a few times after new models appeared, so US models progression was much higher than that of other vendors (like when they updated the algorithm after Kimi K3 versus Opus 4.7, so Kimi dropped in the rankings). Or maybe thats just coincidence. [0] https://artificialanalysis.ai/changelog https://artificialanalysis.ai/changelog

dist-epoch 11d ago
That would be the fastest way for them to completely torch their company. The only thing they are selling and why people look at them is trust that they do honest evaluations.
- marmarama 10d ago
  People still buy into "research" from Gartner and Forrester despite it being an open secret for decades that they are pay-to-play. These kinds of "research" companies are there to validate people's preconceived ideas and purchasing decisions rather than being genuinely unbiased. And they are very useful for that, both for consumers and for marketers.

dgacmu 11d ago

I quite wish they'd move to terminal bench 4.0. 2.1 is saturated - there is no world in which Gemini 3.8 Flash is producing better code than Astra or Fable, as the 2.1 results might suggest. The 4.0 results differentiate these models much more effectively. (2.1 is useful for knowing they can all one-shot straightforward scripts, of course.)

swingboy 11d ago
DeepSWE has Gemini 3.8 Flash up really high, too.
- dgacmu 11d ago
  It does and that one also feels kind of saturated for measuring the most advanced models - opus, Gemini, astra, sol, fable, glm, kimi all scoring within statistical noise of each other. (74 +-3% down to 69% +-5% for kimi). It's still providing strong discrimination between weaker models.

ahmedelsama 11d ago

[flagged]

MoreThanMe 11d ago

[flagged]

snezhadianpm 11d ago

[dead]

theagenticleade 11d ago

Are you sure that Fable 5.1 outperforms Astra? Feel like early indications and testing appear to show for the first time a true Agentic experience? This could open the door to a true loop build in every corner of the world.

sheepscreek 11d ago

Seeing the Qwen3.8-27B model among giants is almost comical. It is unparalleled and a true testament to Alibaba’s talent and ingenuity. Can’t overstate how excited I am for newer edge models.

mmmmbbbhb 10d ago

This website lost all trustworthiness with the astra blunder. The benchmarks are worthless.