9 ms·

I think I'd typify it as "ARC-AGI doesn't matter" more than "harness matters". Or maybe "harness matters for some very specific tasks".

by Zababa 2mo ago

YeGoblynQueenne 2mo ago
ARC-AGI is a bit of a joke at this point. The first version was supposed to be hard for neural nets because it had few examples and a super secret test set, with a different distribution than the public set no less. It took a while, but the original ARC ultimately fell to exactly the approach it was supposed to be protected from, i.e. big data memorisation, thanks to data augmentation techniques that increased the available training examples to the point that the private test set was eventually overcome. ARC-AGI 2 went the same way because it was basically the same kind of dataset except this time with some attempt to further defend it against LLMs with restrictions on the compute budget. And now ARC-AGI 3 is saturated within ... what is it, weeks? since its release. The fact that it's the public set that's beaten doesn't matter, when the score is 99%. Systems that can score ~90% on the public sets of the previous ARC's can comfortably reach 70-80% on the corresponding private test sets, as far as my eyballing of results suggests. It is time to accept that the whole idea of ARC is for the dustbin. It does not measure what it's supposed to measure -fluid intelligence, reasoning, whatever it is today. Its original premise, that a system could only beat ARC if it possessed human-like core knowledge systems (a-la Elizabeth Spelke's theory) has been comprehensively refuted: none of the systems that have ever performed well on any version of ARC has made any attempt to represent core knowledge systems in any way, shape or form. Ultimately, if your machine intelligence (let alone AGI) test relies on tricks like only giving a few examples or keeping a secret test set to defend itself against the dominant approach to machine intelligence... then it's not a useful machine intelligence test. Or it just doesn't measure machine intelligence but... something else. Who knows what.
- Legend2440 2mo ago
  >It took a while, but the original ARC ultimately fell to exactly the approach it was supposed to be protected from, i.e. big data memorisation No it didn't. People tried big data memorization, and it didn't work. Base LLMs (even with millions of synthetic examples) never solved ARC-AGI-1. It took a real algorithmic advancement - reasoning models - to solve it.
  - Zababa 2mo ago
    DeepSeek V3.2 was tried without reasoning and it got 57% on ARC AGI 1. It's a 7 month model, so I'm pretty confident that base LLMs would be able to solve ARC AGI 1 without reasoning/CoT.
    - Legend2440 2mo ago
      No, you are misunderstanding the paper. https://arxiv.org/abs/2607.06764 https://arxiv.org/abs/2607.06764 The base model got 15%. They built an elaborate looping harness that allows it to burn 100k tokens "thinking" about the problem, which got the 57%. This is just an alternative approach to reasoning.
      - Zababa 2mo ago
        I'm not referring to this paper, I'm referring to this leaderboard: https://arcprize.org/leaderboard https://arcprize.org/leaderboard. Set it to "arc agi 1", "base LLM" and you'll see deepseek at 57%. Submitted 2025-12-01, $0.120 per task. The paper you linked was later than that, and also says "We do not report an official ARC Prize leaderboard score". So this paper doubled the price to get the same exact result at base Deepseek 3.2 at launch, and wasn't even tested on the verified set.
        Legend2440 2mo ago
        I think this is an error in the leaderboard. Looking at the test logs, they had reasoning effort set to 'high'. So it should be in the CoT category instead of the base LLM category. https://huggingface.co/datasets/arcprize/arc_agi_v1_public_eval/tree/main/deepseek-v3.2 https://huggingface.co/datasets/arcprize/arc_agi_v1_public_e... "kwargs": { "max_tokens": 100000, "stream": true, "reasoning_effort": "high", "rate_limit": { "rate": 2, "period": 60 } } The other paper ran Deepseek v3.2 without reasoning as a baseline and got 15.5%, which is much more in line with other base LLMs like GPT-5.2.
        2mo ago
- HarHarVeryFunny 2mo ago
  Benchmarks are meant to measure something, so can't be too hard else all measurements will be 0. At the same time the systems being tested - LLMs - are getting larger and more capable, at least in the narrow areas most benchmarks are focusing on, so all benchmarks will continually become saturated and need to be revised. So, are you against all benchmarks or specifically ARC AGI? At least ARC AGI is trying to test for something a bit different and not play to the text prediction strength of LLMs. It should go without saying that no single test, or type of test, can claim to test for AGI or human level intelligence, which would require a suite of tests as broad and varied as the generality of intelligence you are trying to test for.

I think I'd typify it as "ARC-AGI doesn't matter" more than "harness matters". Or maybe "harness matters for some very specific tasks".

by Zababa 2mo ago

I think I'd typify it as "ARC-AGI doesn't matter" more than "harness matters". Or maybe "harness matters for some very specific tasks".

YeGoblynQueenne 2mo ago

ARC-AGI is a bit of a joke at this point. The first version was supposed to be hard for neural nets because it had few examples and a super secret test set, with a different distribution than the public set no less. It took a while, but the original ARC ultimately fell to exactly the approach it was supposed to be protected from, i.e. big data memorisation, thanks to data augmentation techniques that increased the available training examples to the point that the private test set was eventually overcome. ARC-AGI 2 went the same way because it was basically the same kind of dataset except this time with some attempt to further defend it against LLMs with restrictions on the compute budget. And now ARC-AGI 3 is saturated within ... what is it, weeks? since its release. The fact that it's the public set that's beaten doesn't matter, when the score is 99%. Systems that can score ~90% on the public sets of the previous ARC's can comfortably reach 70-80% on the corresponding private test sets, as far as my eyballing of results suggests. It is time to accept that the whole idea of ARC is for the dustbin. It does not measure what it's supposed to measure -fluid intelligence, reasoning, whatever it is today. Its original premise, that a system could only beat ARC if it possessed human-like core knowledge systems (a-la Elizabeth Spelke's theory) has been comprehensively refuted: none of the systems that have ever performed well on any version of ARC has made any attempt to represent core knowledge systems in any way, shape or form. Ultimately, if your machine intelligence (let alone AGI) test relies on tricks like only giving a few examples or keeping a secret test set to defend itself against the dominant approach to machine intelligence... then it's not a useful machine intelligence test. Or it just doesn't measure machine intelligence but... something else. Who knows what.

Legend2440 2mo ago
>It took a while, but the original ARC ultimately fell to exactly the approach it was supposed to be protected from, i.e. big data memorisation No it didn't. People tried big data memorization, and it didn't work. Base LLMs (even with millions of synthetic examples) never solved ARC-AGI-1. It took a real algorithmic advancement - reasoning models - to solve it.
- Zababa 2mo ago
  DeepSeek V3.2 was tried without reasoning and it got 57% on ARC AGI 1. It's a 7 month model, so I'm pretty confident that base LLMs would be able to solve ARC AGI 1 without reasoning/CoT.
  - Legend2440 2mo ago
    No, you are misunderstanding the paper. https://arxiv.org/abs/2607.06764 https://arxiv.org/abs/2607.06764 The base model got 15%. They built an elaborate looping harness that allows it to burn 100k tokens "thinking" about the problem, which got the 57%. This is just an alternative approach to reasoning.
    - Zababa 2mo ago
      I'm not referring to this paper, I'm referring to this leaderboard: https://arcprize.org/leaderboard https://arcprize.org/leaderboard. Set it to "arc agi 1", "base LLM" and you'll see deepseek at 57%. Submitted 2025-12-01, $0.120 per task. The paper you linked was later than that, and also says "We do not report an official ARC Prize leaderboard score". So this paper doubled the price to get the same exact result at base Deepseek 3.2 at launch, and wasn't even tested on the verified set.
      - Legend2440 2mo ago
        I think this is an error in the leaderboard. Looking at the test logs, they had reasoning effort set to 'high'. So it should be in the CoT category instead of the base LLM category. https://huggingface.co/datasets/arcprize/arc_agi_v1_public_eval/tree/main/deepseek-v3.2 https://huggingface.co/datasets/arcprize/arc_agi_v1_public_e... "kwargs": { "max_tokens": 100000, "stream": true, "reasoning_effort": "high", "rate_limit": { "rate": 2, "period": 60 } } The other paper ran Deepseek v3.2 without reasoning as a baseline and got 15.5%, which is much more in line with other base LLMs like GPT-5.2.
        2mo ago
HarHarVeryFunny 2mo ago
Benchmarks are meant to measure something, so can't be too hard else all measurements will be 0. At the same time the systems being tested - LLMs - are getting larger and more capable, at least in the narrow areas most benchmarks are focusing on, so all benchmarks will continually become saturated and need to be revised. So, are you against all benchmarks or specifically ARC AGI? At least ARC AGI is trying to test for something a bit different and not play to the text prediction strength of LLMs. It should go without saying that no single test, or type of test, can claim to test for AGI or human level intelligence, which would require a suite of tests as broad and varied as the generality of intelligence you are trying to test for.