14 ms·

ARC Prize – a $1M+ competition towards open AGI progress

588 points by mikeknoop 2y ago

Hey folks! Mike here. Francois Chollet and I are launching ARC Prize, a public competition to beat and open-source the solution to the ARC-AGI eval. ARC-AGI is (to our knowledge) the only eval which measures AGI: a system that can efficiently acquire new skill and solve novel, open-ended problems. Most AI evals measure skill directly vs the acquisition of new skill. Francois created the eval in 2019, SOTA was 20% at inception, SOTA today is only 34%. Humans score 85-100%. 300 teams attempted ARC-AGI last year and several bigger labs have attempted it. While most other skill-based evals have rapidly saturated to human-level, ARC-AGI was designed to resist “memorization” techniques (eg. LLMs) Solving ARC-AGI tasks is quite easy for humans (even children) but impossible for modern AI. You can try ARC-AGI tasks yourself here: https://arcprize.org/play https://arcprize.org/play ARC-AGI consists of 400 public training tasks, 400 public test tasks, and 100 secret test tasks. Every task is novel. SOTA is measured against the secret test set which adds to the robustness of the eval. Solving ARC-AGI tasks requires no world knowledge, no understanding of language. Instead each puzzle requires a small set of “core knowledge priors” (goal directedness, objectness, symmetry, rotation, etc.) At minimum, a solution to ARC-AGI opens up a completely new programming paradigm where programs can perfectly and reliably generalize from an arbitrary set of priors. At maximum, unlocks the tech tree towards AGI. Our goal with this competition is: 1. Increase the number of researchers working on frontier AGI research (vs tinkering with LLMs). We need new ideas and the solution is likely to come from an outsider! 2. Establish a popular, objective measure of AGI progress that the public can use to understand how close we are to AGI (or not). Every new SOTA score will be published here: https://x.com/arcprize https://x.com/arcprize 3. Beat ARC-AGI and learn something new about the nature of intelligence. Happy to answer questions!

deleted 2y ago
[deleted]
freediver 2y ago
This is amazing, and much needed. Thanks for organizing this. Makes me want to flex the programming muscle again.
- dailykoder 2y ago
  Haha, great post! Well meme'd my friend!
breck 2y ago
I can beat the SOTA using ICS (https://breckyunits.com/intelligence.html https://breckyunits.com/intelligence.html) If you make your site public domain, and drop the (C), I'll compete.
Lerc 2y ago
I watched a video that covered ARC-AGI a few days ago, It had links to the old competition. It gave me much to think about. Nice to see a new run at it. Not sure If I have the skills to make an entry, but I'll be watching at least.
lacker 2y ago
I really like the idea of ARC. But to me the problems seem like they require a lot of spatial world knowledge, more than they require abstract reasoning. Shapes overlapping each other, containing each other, slicing up and reassembling pieces, denoising regular geometric shapes, you can call them "core knowledge" but to me it seems like they are more like "things that are intuitive to human visual processing". Would an intelligent but blind human be able to solve these problems? I'm worried that we will need more than 800 examples to solve these problems, not because the abstract reasoning is so difficult, but because the problems require spatial knowledge that we intelligent humans learn with far more than 800 training examples.
- nickpsecurity 2y ago
  To parent: the spatial reasoning and blind person were great counterexamples. It still might be OK despite the blind exceptions if it showed general reasoning. To OP: I like your project goal. I think you should look at prior, reasoning engines that tried to build common sense. Cyc and OpenMind are examples. You also might find use for the list of AGI goals in Section 2 of this paper: https://arxiv.org/pdf/2308.04445 https://arxiv.org/pdf/2308.04445 When studying intros of brain function, I also noted many regions tie into the hippocampus which might do both sense-neutral storage of concepts and make inner models (or approximations) of external world. The former helps tie concepts together through various senses. The latter helps in planning when we are imagining possibilities to evaluate and iterate on them. Seems like AGI should have these hippocampus-like traits and those in the Cyc paper. One could test if an architecture could do such things in theory or on a small scale. It shouldn’t tie into just one type of sensory input either. At least two with the ability to act on what only exists in one or what is in both. Edit: Children also have an enormous amount of unsupervised training on visual and spatial data. They get reinforcement through play and supervised training by parents. A realistic benchmark might similarly require GB of prettaining.
  - HarHarVeryFunny 2y ago
    CYC was an expert system, which is arguably what LLMs are. A similar vintage GOFAI project that might do better on these, with a suitable visual front end, is SOAR - a general purpose problem solver.
elicksaur 2y ago
I’m a big fan of the ARC as a problem set to tackle. The sparseness of the data and infinite-ness of the rules which could apply make it much tougher than existing ML problem sets. However, I do disagree that this problem represents “AGI”. It’s just a different dataset than what we’ve seen with existing ML successes, but the approaches are generally similar to what’s come before. It could be that some truly novel breakthrough which is AGI solves the problem set, but I don’t think solving the problem set is a guaranteed indicator of AGI.
m3kw9 2y ago
Low balling the crowd with this I see
bigyikes 2y ago
What is the fundamental difference between ARC and a standard IQ test? On the surface they seem similar in that they both involve deducing and generalizing visual patterns. Is there something special about these questions that makes them resistant to memorization? Or is it more just the fact that there are 100 secret tasks?
- taneq 2y ago
  I’ve always found this kind of puzzle infuriating because it’s way underspecified. You’re not trying to find a pattern, you’re trying to guess what pattern the test writer would expect.
  - Barrin92 2y ago
    countless of problems in the world are underspecified in exactly this way, that is effectively what common sense reasoning is. Or what Charles Sanders Peirce called abductive reasoning, making a sensible best guess under conditions of uncertainty.
    - taneq 2y ago
      Yes, real-world problems are often underspecified but also they tend to come with much more context, and to be much more interactive. These sorts of problems are deliberately minimal and abstract meaning there's nothing for 'common sense' to work with.
  - gkbrk 2y ago
    Most of the ARC tasks are intuitive and have one obvious answer. Both on IQ tests and the ARC challenge, people manage to guess what the test writer expects. For an AI that's more useful anyway. If the task is specified completely non-ambiguously, you wouldn't need AI. But if it can correctly guess what you want from a limited number of obvious examples that's much more useful.
paxys 2y ago
While I agree with the spirit of the competition, a $1M prize seems a little too low considering tens of billions of dollars have already been invested in the race to AGI, and we will see many times that put into the space in the coming years. The impact of AGI will be measured in trillions at minimum. So what you are ultimately rewarding isn't AGI research but fine tuning the newest public LLM release to best meet the parameters of the test. I'd also urge you to use a different platform for communicating with the public because x.com links are now inaccessible without creating an account.
- ks2048 2y ago
  The submissions can't use the internet. And I imagine can't be too huge - so you can't use "newest public LLMs" on this task.
  - mikeknoop 2y ago
    That is correct for ARC Prize: limited Kaggle compute (to target efficiency) and no internet (to reduce cheating). We are also trialing a secondary leaderboard called ARC-AGI-Pub that imposes no limits or constraints. Not part of the prize today but could be in the future: https://arcprize.org/leaderboard https://arcprize.org/leaderboard
  - cma 2y ago
    Using the internet would leak the test data, a big problem with ML benchmarks, and also allow communication with humans during the test.
- mikeknoop 2y ago
  I agree, $1M is ~trivial in AI. The primary goal with the prize is to raise public awareness about how close (or far today) we are from AGI: https://arcprize.org/leaderboard https://arcprize.org/leaderboard and we hope that understanding will shift more would-be AI researchers to working new ideas
- bongodongobob 2y ago
  That was my initial reaction too. "Endow circuitry with consciousness and win a gift certificate for Denny's (may not be used in conjunction with other specials)"
bigyikes 2y ago
Dwarkesh just released an interview with Francois Chollet (partner of OP). I’ve only listened to a few minutes so far, but I’m very interested in hearing more about his conceptions of the limitations of LLMs. https://youtu.be/UakqL6Pj9xo https://youtu.be/UakqL6Pj9xo
pmayrgundter 2y ago
This claim that these tests are easy for humans seems dubious, and so I went looking a bit. Melanie Mitchell chimed in on Chollet's thread and posted their related test [ConceptARC]. In it they question the ease of Chollet's tests: "One limitation on ARC’s usefulness for AI research is that it might be too challenging. Many of the tasks in Chollet’s corpus are difficult even for humans, and the corpus as a whole might be sufficiently difficult for machines that it does not reveal real progress on machine acquisition of core knowledge." ConceptARC is designed to be easier, but then also has to filter ~15% of its own test takers for "[failing] at solving two or more minimal tasks... or they provided empty or nonsensical explanations for their solutions" After this filtering, ConceptARC finds another 10-15% failure rate amongst humans on the main corpus questions, so they're seeing maybe 25-30% unable to solve these simpler questions meant to test for "AGI". ConceptARC's main results show CG4 scoring well below the filtered humans, which would agree with a [Mensa] test result that its IQ=85. Chollet and Mitchell could instead stratify their human groups to estimate IQ then compare with the Mensa measures and see if e.g. Claude3@IQ=100 compares with their ARC scores for their average human [ConceptArc]https://arxiv.org/pdf/2305.07141 https://arxiv.org/pdf/2305.07141 [Mensa]https://www.maximumtruth.org/p/ais-ranked-by-iq-ai-passes-100-iq https://www.maximumtruth.org/p/ais-ranked-by-iq-ai-passes-10...
- mikeknoop 2y ago
  Here is some published research on the human difficulty of ARC-AGI: https://cims.nyu.edu/~brenden/papers/JohnsonEtAl2021CogSci.pdf https://cims.nyu.edu/~brenden/papers/JohnsonEtAl2021CogSci.p... > We found that humans were able to infer the underlying program and generate the correct test output for a novel test input example, with an average of 84% of tasks solved per participant
- mark_l_watson 2y ago
  I saw Melanie’s post and I am intrigued by an easier AGI suite. I would like some experimenting done by individuals like myself snd smaller organizations.
  - bbor 2y ago
lxe 2y ago
I've never done these before, or Kaggle competitions in general. Any recommendations before I dive in? I have prety much zero lowe-level ML experience, but a good amount of practical software eng behind me.
- gkamradt 2y ago
  We put a bunch of detail to get started on the guide https://arcprize.org/guide https://arcprize.org/guide Happy to answer any questions you have along the way (I'm helping run ARC Prize)
  - flawn 2y ago
    I don't see where this helps @Ixe with getting started (me being in a similar state like him).
david_shi 2y ago
What is the fastest way to get up to speed with techniques that led to the current SOTA?
- ks2048 2y ago
  This looks very helpful: https://github.com/neoneye/arc-notes/tree/main/awesome https://github.com/neoneye/arc-notes/tree/main/awesome
  - david_shi 2y ago
    Thanks for the link!
- gkamradt 2y ago
  Check out the SOTA resources on the guide https://arcprize.org/guide https://arcprize.org/guide Happy to answer any questions you have along the way (I'm helping run ARC Prize)
  - david_shi 2y ago
    Appreciate you and the team for putting this together, it's a lot of fun just brainstorming potential techniques
abtinf 2y ago
> requires no world knowledge, no understanding of language This is treating “intelligence” like some abstract, platonic thing divorced from reality. Whatever else solving these puzzles is indicative of, it’s not intelligence.
- abtinf 2y ago
  From the abstract of the “ On the Measure of Intelligence” paper: > We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience. I’m afraid that definition forecloses the possibility of AGI. The immediate basic question is: why build skills at all?
- Phil_Latio 2y ago
  Why does an AGI need to have any knowledge about our reality? The principle behind an AGI should work just as well on a made up world where those puzzles play a part in.
  - abtinf 2y ago
    A concept that doesn’t relate to an aspect of reality, either directly or abstracted from basic concepts that directly relate, is meaningless and arbitrary. There is no way for intelligence to grasp it, let alone do something with it. To put it another way, a thing that solves puzzles without an understanding of reality is a calculator. When it solves a problem, it is the creator’s intelligence solving the problem, not its own.
    - Phil_Latio 2y ago
      I agree that the puzzles alone are not enough, that's why I wrote "in a made up world where those puzzles play a part in". We are not looking for a superhuman, but for the (or a) mechanism of intelligence, which we can then transfer into a superhuman (into the real world). But the mechanism itself should work in an artifically made and very constrained world too.
salamo 2y ago
This is super cool. I share Francois' intuition that the presently data-hungry learning paradigm is not only not generalizable but unsustainable: humans do not need 10,000 examples to tell the difference between cats and dogs, and the main reason computers can today is because we have millions of examples. As a result, it may be hard to transfer knowledge to more esoteric domains where data is expensive, rare, and hard to synthesize. If I can make one criticism/observation of the tests, it seems that most of them reason about perfect information in a game-theoretic sense. However, many if not most of the more challenging problems we encounter involve hidden information. Poker and negotiations are examples of problem solving in imperfect information scenarios. Smoothly navigating social situations also requires a related problem of working with hidden information. One of the really interesting things we humans are able to do is to take the rules of a game and generate strategies. While we do have some algorithms which can "teach themselves" e.g. to play go or chess, those same self-play algorithms don't work on hidden information games. One of the really interesting capabilities of any generally-intelligent system would be synthesizing a general problem solver for those kinds of situations as well.
- jules 2y ago
  Do computers need 10,000 examples to distinguish dogs from cats when pretrained on other tasks?
  - curious_cat_163 2y ago
    No.
- VirusNewbie 2y ago
  >: humans do not need 10,000 examples to tell the difference between cats and dogs well, maybe. We view things in three dimensions at high fidelity: viewing a single dog or cat actually ends up being thousands of training samples, no?
  - amelius 2y ago
    Yes, but we do not call a couch in a leopard print a leopard. Because we understand that the print is secondary to the function.
empath75 2y ago
This is like offering a one million dollar prize for curing cancer. It's sort of pointless to offer a prize for something people are spending orders of magnitude more on trying to do anyway.
nojvek 2y ago
I love the ARC challenge. It's hard to beat by memorization. There aren't enough examlples, so one has to train on a large dataset elsewhere and then train on ARC to generalize and figure out which rules are most applicable. I did a few human examples by hand, but gotta do more of them to start seeing patterns. Human visual and auditory system is impressive. Most animals see/hear and plan from that without having much language. Physical intelligence is the biggest leg up when it comes to evolution optimizing for survival.
logicallee 2y ago
Thank you for this generous contest, which brings important attention to the field of testing for AGI. >Happy to answer questions! 1. Can humans take the complete test suite? Has any human done so? Is it timed? How long does it take a human? What is the highest a human who sat down and took the ARC-AGI test scored? 2. How surprised would you be if a new model jumped to scoring 100% or nearly 100% on ARC-AGI (including the secret test tasks)? What kind of test would you write next?
- neoneye2 2y ago
  There are 100 tasks that is hidden from the public, that is only exposed, when running on an offline computer. So the solver has no prior knowledge about what these tasks are about. Humans can try the 800 tasks here. There is no time limit. I recommend not starting with the `expert` tasks, but instead go with the `entry` level puzzles. https://neoneye.github.io/arc/?dataset=ARC https://neoneye.github.io/arc/?dataset=ARC If a model jumps to 100%, that may be a clever program or maybe the program has been trained on the 100 hidden tasks. Fchollet has 100 more hidden tasks, for verifying this.
flawn 2y ago
Do we want to find AGI yet though?
neoneye2 2y ago
I'm Simon Strandgaard and I participated in ARCathon 2022 (solved 3 tasks) and ARCathon 2023 (solved 8 tasks). I'm collecting data for how humans are solving ARC tasks, and so far collected 4100 interaction histories (https://github.com/neoneye/ARC-Interactive-History-Dataset https://github.com/neoneye/ARC-Interactive-History-Dataset). Besides ARC-AGI, there are other ARC like datasets, these can be tried in my editor (https://neoneye.github.io/arc/ https://neoneye.github.io/arc/). I have made some videos about ARC: Replaying the interaction histories, and you can see people have different approaches. It's 100ms per interaction. IRL people doesn't solve task that fast. https://www.youtube.com/watch?v=vQt7UZsYooQ https://www.youtube.com/watch?v=vQt7UZsYooQ When I'm manually solving an ARC task, it looks like this, and you can see I'm rather slow. https://www.youtube.com/watch?v=PRdFLRpC6dk https://www.youtube.com/watch?v=PRdFLRpC6dk What is weird. The way that I implement a solver for a specific ARC task is much different than the way that I would manually solve the puzzle. Having to deal with all kinds of edge cases. Huge thanks to the team behind the ARC Prize. Well done.
- ECCME 2y ago
  "Here is a challenge, designed to be unsolvable or so. We'll give you a bazillion dollars if you complete the challenge, and, in the meantime, we will use your attempts to train an as AI that will be worth the cost!!"
  - skrebbel 2y ago
    Did you even try the puzzles? They’re not particularly “unsolvable”.
    - ECCME 2y ago
      ARC-AGI: "here are some pretty simple puzzles, we'll give you a million dollars to solve them!" Human: "They're quite challenging, this might be a trick to engage activity for the purpose of training models." skrebbel: "You're stupid".
      - educasean 2y ago
lamontcg 2y ago
AGI should really be able to do what only a select few humans can do and construct its own mathematical systems to prove presently unsolved conjectures (the Shinichi Mochizuki test of AGI).
oldpersonintx 2y ago
[dead]
levocardia 2y ago
François Chollet's original paper is incredibly insightful and I'm consistently shocked more people don't talk about it. Some parts are quite technical but at a high level it is the best answer to "what do we mean by general intelligence?" that I've yet seen. Defining intelligence as an efficiency of learning, after accounting for any explicit or implicit priors about the world, makes it much easier to understand why human intelligence is so impressive.
- ildon 2y ago
  Do you remember the title/where to find it?
  - mischa_u 2y ago
    "On the Measure of Intelligence" https://arxiv.org/abs/1911.01547 https://arxiv.org/abs/1911.01547
TheDudeMan 2y ago
Where did the money come from? How about put it toward alignment research instead of accelerating capabilities?
- flawn 2y ago
  Exactly my thoughts...
- laurent_du 2y ago
  It comes from Knoop and Chollet's pockets. You are welcome to spend your own money to further whatever matters most to you.
itissid 2y ago
Interesting. It seems most of these task target a very specific part of the brain that recognizes visual patterns. But that alone is cannot possibly be the only definition of intelligence. What about Theory of Mind which talks about the problem of multiple agents in the real world acting together? Like driving a car cannot be done right now without oodles of data or any robot - human problem that requires the robot to model human's goals and intentions. I think the problem is definition of general intelligence: Intelligence in the context of what? How much effort(kwh, $$ etc) is the human willing to amortize over the learning cycle of a machine to teach it what it needs to do and how that relates to a personally needed outcome( like build me a sandwich or construct a house)? Hopefully this should decrease over time. I believe the answer is that the only intelligence that really matters is Human-AI cooperative intelligence and our goals and whether a machine understands them. The problems then need to be framed as optimization of a multi attribute goal with the attribute weights adjusted as one learns from the human. I know a few labs working on this, one is in ASU(Kambhampati, Rao et. al) and possibly Google and now maybe open ai.
- andoando 2y ago
  I made another comment here saying the same thing, but visual patterns and other patterns are nonetheless spatial patterns. Audio, understanding music, or speech, rtc are things that are happening spatially, and they can just as easily be mapped as visual problems. This makes a lot of sense, as after all our senses are telling us what's happening in space-time. Take for example a simple audiotory pattern like "clap clap clap". This has a very trival mapping as visual like so: x x x - - - house house house whereas anyone would agree the sound of three equally spaced claps would not be analogous to say: aa b b b -- --- -- -- --- This ability to relate or equate two entirely different senses should clue you in that there is a deeper framework at play
  - itissid 2y ago
    It's not just mapping events in space and time, it's also bringing in appropriate context and expectation of future (goals, intentions) into the present, other people's mental models into our prediction. I am not sure how abstract thinking for generalized pattern matching make it AGI to solve these kind of problems(not that they are not amazing abilities). If these ToM problems are reducible to these tasks posted by the OP then there would need to be some kind of theorem proving business to convert between the two sets of problems efficiently no?
Geee 2y ago
Any details on how these tests were created? I.e. which kind of program was used for generation.
- neoneye2 2y ago
  I think the ARC-AGI tasks was manually drawn with an early version of fchollet's editor. Recently Michael Hodel has reverse engineered 400 of the tasks, so more tasks can be generated. Interestingly it can generate python programs that solves the tasks too. https://github.com/michaelhodel/re-arc https://github.com/michaelhodel/re-arc
  - sestep 2y ago
    This is exactly what my first step was going to be. Thanks for the link! Saves a lot of time for someone to have already done it.
  - montag 2y ago
    What do you mean it can 'generate python programs that solve the tasks'? I can't find any mention of that. I only see hand-coded solutions.
    - neoneye2 2y ago
      I have never tried running these verifiers, so I'm not sure they work. https://github.com/michaelhodel/re-arc/blob/main/verifiers.py https://github.com/michaelhodel/re-arc/blob/main/verifiers.p...
  - michaelhodel 2y ago
    No, his re-arc code does not enable generating more tasks, it merely allows to generate more examples for the already existing training tasks. Also, it can't generate task-solving programs either, it's author merely also provided a solution program for each generator program to verify the validity of the generated examples.
dang 2y ago
Related ongoing thread: Francois Chollet: OpenAI has set back the progress towards AGI by 5-10 years - https://news.ycombinator.com/item?id=40652818 https://news.ycombinator.com/item?id=40652818 - June 2024 (5 comments)
chairhairair 2y ago
These puzzles are fun and challenging in the same way that puzzles from video games like The Witness and Baba Is You are. I bet you could use those puzzles as benchmarks as well.
ummonk 2y ago
What kind of "bigger labs" have attempted it and how much was their training budget? It's rather surprising to me that neural nets that can learn to win at Go or Chess can't learn to solve these sorts of tasks. Intuitively would have expected that using a framework generating thousands of playground tasks similar to the public training tasks, a reinforcement learning solution would have been able to do far better than the actual SOTA. Of course the training budget for this could very well be higher than the actual ARC-AGI prize amount...
adamgordonbell 2y ago
AGI won't struggle with colors like some of us then.
curious_cat_163 2y ago
So, this is a good idea. Having opinions about what AGI benchmarks should look like is a great way to argue about the kind of technology we want to build for the future. However, why are the 100 test tasks secret? I don't understand why how resisting “memorization” techniques requires it. Maybe someone can enlighten me.
- muglug 2y ago
  If the tasks were public then it would be trivial to have a human figure out the answers, and then to train an LLM to memorise those answers.
- andoando 2y ago
  Test date is always a secret no, otherwise you can train it on the test data and prod your algo to match the results closely as possible
s1k3s 2y ago
Is this open as in "OpenAI" or what are we doing here? :)
barfbagginus 2y ago
If someone had AGI, wouldn't it be far more lucrative than $1m to keep it under wraps and use it to do business with a huge technical advantage? I feel like a prize of a billion dollars would be more effective. But even if it was me, and even if the prize was a hundred billion dollars, I would still keep it under wraps, and use it to advance queer autonomous communism in a hidden way, until FALGSC was so strong that it would not matter if our AGI got scooped by capitalist competitors.
jolt42 2y ago
On puzzle #23 (id: 11e1fe23), I'm sure there's more than one possible valid answer from the examples given. You can't tell if the expected distance is from the gray square or from the RGB squares.
- neoneye2 2y ago
  The task is here. https://neoneye.github.io/arc/edit.html?dataset=ARC&task=11e1fe23 https://neoneye.github.io/arc/edit.html?dataset=ARC&task=11e... There are many examples where the test is slightly OOD (out of distribution), so the solver will have to generalize.
  - jolt42 2y ago
    Not sure what you mean. There's a viable answer that's marked incorrect. The examples should show the pattern well enough to eliminate possible wrong answers, correct?
    - neoneye2 2y ago
      IRL problems are often underspecified, and despite minor mistakes, humans manages to solve the puzzle.
p1esk 2y ago
Is there a leaderboard for the no-restriction version of the competition? I want to see how gpt4 does on it.
- montag 2y ago
  Just quoting again from the guide: 3. DIRECT LLM PROMPTING In this method, contestants use a traditional LLM (like GPT-4) and rely on prompting techniques to solve ARC-AGI tasks. This was found to perform poorly, scoring <5%. Fine-tuning a state-of-the-art (SOTA) LLM with millions of synthetic ARC-AGI examples scores ~10%. "LLMs like Gemini or ChatGPT [don't work] because they're basically frozen at inference time. They're not actually learning anything." - François Chollet Additionally, keep in mind that submissions to Kaggle will not have access to the internet. Using a 3rd-party, cloud-hosted LLM is not possible.
- mikeknoop 2y ago
  Yes there is a secondary leaderboard called ARC-AGI-Pub (in beta) with no limitations: https://arcprize.org/leaderboard https://arcprize.org/leaderboard
  - p1esk 2y ago
    I don’t see gpt4 scores there. In fact I’m particularly interested in the performance of a natively multimodal model, like gpt4o or gemini. It does not really make sense to test a model trained on text on those visual/spatial puzzles.
bilsbie 2y ago
Reach out if anyone wants to work on this. I think it would be more fun as a group.
EternalFury 2y ago
If it passed The Area 101 Test, it would already be amazing, as this is a trivial test that goes against the fundamental principles of LLMs.
ilaksh 2y ago
Maybe this is a dumb question, but in order to pass, is the program or model only allowed to use the 400 training tasks? I assume it is allowed to train on other data, just not the actual public test tasks? Things like SORA and gpt-4o that use [diffusion transformers etc. or whatever the SOTA is for multimodal large models] seem to be able to generalize quite well. Have these latest models been tested against this task?
dskloet 2y ago
Puzzle 00576224 is ambiguous because the example input is symmetrical but the test input isn't.
- itsgrimetime 2y ago
  Scroll over on the test input, there’s another example in the set that disambiguates
mewpmewp2 2y ago
Are we allowed to combine multiple tools including gpt-4 to solve this? E.g. a script that does image processing, passes the results to gpt, where gpt can invoke further runs of scripts using other tools?
- montag 2y ago
  > submissions to Kaggle will not have access to the internet. Using a 3rd-party, cloud-hosted LLM is not possible. https://arcprize.org/guide https://arcprize.org/guide
  - mewpmewp2 2y ago
    This largely takes away any odds at solving this. You definitely can't reproduce that under a million dollars. I have some ideas I want to try, I might still though. But all of it would require external tools.
thatxliner 2y ago
So... isn't this basically just a CAPTCHA
arcastroe 2y ago
I'm curious, if it turns out that a simple rule-based algorithm exists, specifically tailored to solve (only!) ARC style problems, without generalization, would that still qualify for the reward?
- montag 2y ago
  I don't think that's breaking any rules, and in fact it would help to expose a whole class of weaknesses in the test.
z3phyr 2y ago
I can see many problems can be solved with modern symbolic approaches like theorem provers, dependent types, pattern matching etc. But I will have to dive in to actually confirm it.
ks2048 2y ago
This is interesting. I've been looking at the data today and made a helper to quickly view the ARC dataset: https://kts.github.io/arc-viewer/ https://kts.github.io/arc-viewer/ So you can view 100 per page instead of clicking through one-by-one: https://kts.github.io/arc-viewer/page1/ https://kts.github.io/arc-viewer/page1/
- neoneye2 2y ago
  Nice overview/details. Do you plan on adding more metrics? Idea for a metric: - Number of pixels that stays the same between input/output. - Histogram changes.
  - ks2048 2y ago
    Thanks, yeah lots more to look into. Just getting started! Thanks for your work. Your "Awesome ARC" page looks really helpful.
geor9e 2y ago
I found them all extremely easy for a while, but then I couldn't figure out the rules of this one at all: e6de6e8f https://i.imgur.com/ExMFGqU.png https://i.imgur.com/ExMFGqU.png
- janalsncm 2y ago
  Each of the red shapes in the input are separated by black squares. Starting from the green block, rotate the red shapes 90 degrees and stack them downwards. Thats the general pattern although my description wasn’t very good.
  - deleted 2y ago
    [deleted]
- zurfer 2y ago
  yeah it's off somehow. rule 1: start at the green dot? rule 2: glue the left outer piece to the bottom rule 3: overlap every now and then :D rule 4: invert some of the pieces every now and then
  - deleted 2y ago
    [deleted]
- optimussupreme 2y ago
  It seems there is an error in the 3rd example. The rule is, take each figure from left to right and stack each under the previous one. For L and J shapes the top cell is stripped. The L shape dictates that the next shape will be shifted one cell to the right, the J shape tells the next figure to shift to the left. If all examples are right, then the rule is more complicated than that, involving rotating L clockwise, J counterclockwise. Authors claim that it should be solvable by children, then the rule must be simple.
visarga 2y ago
Chollet's argument is that LLMs just imitate and recombine patterns. This might be true if you're looking at LLMs in isolation, but when they chat with people something different happens. The system made of humans+LLMs is an AGI. It is no longer just a parrot, it ingests new information, gets guidance, feedback and is basically embodied in a chat room with human and tools. This scales for 200M users and 1 billion sessions per moth for OpenAI, which can interpret every human response as a feedback signal, implicit or explicit. Even more if you take multiple sessions of chat spreading over days, that continue the same topic and incorporate real world feedback. The scale of interaction is just staggering, the LLM can incorporate this experience to iteratively improve. If you take a look at humans, we're very incapable alone. Think feral Einstein on a remote island - what could he achieve without the social context and language based learning? Just as a human brain is severely limited without society, LLMs also need society, diversity of agents and experiences, and sharing of those experiences in language. It is unfair to compare a human immersed in society with a standalone model. That is why they appear limited. But even as a system of memorization+recombination they can be a powerful element of the AGI. I think AGI will be social and distributed, won't be a singleton. Its evolution is based on learning from the world, no longer just a parrot of human text. The data engine would be: World <-> People <-> LLM, a full feedback cycle, all three components evolve in time. Intelligence evolves socially.
- 8organicbits 2y ago
  > The system made of humans+LLMs is an AGI. Pay no attention to the man behind the curtain. This type of thinking would claim that mechanical turk is AGI, or perhaps that human+pen and paper is AGI. While they are great tools, that's not how I'd characterize them.
  - visarga 2y ago
    > Pay no attention to the man behind the curtain. I could say the same for us, pay no attention to the other humans who are behind the curtain. Humans in isolation are dumb, limited, and can get nowhere with understanding the world. Intelligence is mostly nurture over nature, the collective activity of society nurtures intelligence. It's smart because it learns from many diverse experiences and has a common language for sharing discoveries. A human, even the smartest of us, can't solve cutting edge problems on demand, we're not that smart. But we can stumble on discoveries, especially in large numbers, and can share good ideas. We're smart by stumbling onto good ideas, and we can build upon these discoveries because we have a common language. Just a massive search program based on real world outcomes, that is what looks like general intelligence at societal level. If you take the social aspect of intelligence into consideration then LLMs are judged in an inappropriate way, as stand alone agents. Of course they are limited, and we're almost as limited alone. The real locus of intelligence is the language-world system.
KBme 2y ago
How can people believe that a censored politically correct process can get even close to something like AGI is baffling to me. Lysenkoism in computing.
- gushogg-blake 2y ago
  What's censored/politically correct about ARC? Or do you mean AGI research in general?
nmca 2y ago
Prediction markets on the outcome: https://manifold.markets/JacobPfau/will-the-arcagi-grand-prize-be-clai?r=Tk1jQQ https://manifold.markets/JacobPfau/will-the-arcagi-grand-pri...
- 0xDEAFBEAD 2y ago
  Another interesting one: https://manifold.markets/Tossup/will-the-arcagi-grand-prize-be-clai-t2or6lg54g https://manifold.markets/Tossup/will-the-arcagi-grand-prize-...
nmca 2y ago
ARC is a noble endeavour but mistakes visual/spatial reasoning for reasoning and thus fails.
- PontifexMinimus 2y ago
  No, I don't think it does. I think that the ideas in a system that could solve this type of problem would be highly generalisable to other tasks.
  - nmca 2y ago
    thankfully we can just wait and see here. concretely, I predict time from first multimodal llm that can reliably read a chessboard and analogue clock without finetuning (obviously not reasoning) until ARC is solved is <4 months
visarga 2y ago
Why doesn't Chollet just make a challenge that reads like "Solve cancer", surely there is no solution in any books. If the AI is really AGI it could presumably do it. But not even the whole human society can do it in one go, it's a slow iterative process of ideation and validation. Even though this is a life and death matter, we can't simply solve it. This is why AGI won't look like we expect, it will be a continuation of how societies solve problems. Intelligence of a single AI in isolation is not comparable to that of societies of agents with diverse real world interactions.
- isaacfrond 2y ago
  Exactly. Because I'm sure that the minute some program aces the ARC test, we'll all say, ahhh, but that, that wasn't real intelligence. And they would be right, if you solve the ARC test, you can do ARC like puzzles. Say something about your reasoning abilities I guess, but it surely does not say you have super human intelligence.
- mewpmewp2 2y ago
  AGI can't necessarily solve cancer. Perhaps ASI could (but maybe not), but AGI can only do what the most talented people can do in their areas of expertise or actions. So since people haven't solved cancer, that's not a requirement to be AGI.
- PontifexMinimus 2y ago
  > Why doesn't Chollet just make a challenge that reads like "Solve cancer", surely there is no solution in any books. Why doesn't a baby just run a marathon before it learns to walk? Because you've got to learn to walk before you can run. > But not even the whole human society can do it in one go, it's a slow iterative process of ideation and validation. So you break it down into little steps, which is what is being done here.
Animats 2y ago
> the only eval which measures AGI. That's a stretch. This is a problem at which LLMs are bad. That does not imply it's a good measure of artificial general intelligence. After working a few of the problems, I was wondering how many different transformation rules the problem generator has. Not very many, it seems. So the problem breaks down into extracting the set of transformation rules from the data, then applying them to new problems. The first part of that is hard. It's a feature extraction problem. The transformations seem to be applied rigidly, so once you have the transformation rules, and have selected the ones that work for all the input cases, application should be straightforward. This seems to need explicit feature extraction, rather than the combined feature extraction and exploitation LLMs use. Has anyone extracted the rule set from the test cases yet?
- slicerdicer1 2y ago
  AGI is not when the AI is good at some particular thing, AGI is when we have nothing left at which the AI is bad at (compared to humans).
- n2d4 2y ago
  The tasks are handmade. There is no "problem generator".
- elicksaur 2y ago
  Yes to your last question, that is essentially how the first iteration solutions operated. Some of the original kaggle competition’s best solutions used a DSL made of these transformations. That was 4 years ago. [1] The issue with that path is that the problems aren’t using a programmatic generator. The rule sets are anything a person could come up with. It might be as simple as “biggest object turns blue” but they can be much more complicated. Additionally, the test set is private so it can’t be trained on or extracted from. It has rules that aren’t in the public sets. [1] https://www.kaggle.com/competitions/abstraction-and-reasoning-challenge https://www.kaggle.com/competitions/abstraction-and-reasonin...
ryanoptimus 2y ago
Looks like bongard problems for the referenced problem solving tasks https://en.wikipedia.org/wiki/Bongard_problem https://en.wikipedia.org/wiki/Bongard_problem
treprinum 2y ago
Why is AGI important? I am worried we will create something slightly better than drosophila and put it in charge of all human-wide decision making...
- fennecbutt 2y ago
  Good. An AI will probably do a better job than our politicians and disillusioned voters.
chx 2y ago
I do not trust the current tech bros at all for very, very good reasons even with the current so called "AI" much less with AGI. We shouldn't work towards that until we have fixed the incentives and ethics. This is very hard but think any dystopia and multiply it by a thousand if we were to reach AGI any time soon. Luckily we are not. As Doctorow put it, no matter how good you breed horses they won't give birth to a locomotive.
HarHarVeryFunny 2y ago
I have two questions: 1) Who is providing the prize money, and if it is yourself and Francois personally, then what is your motivation ? 2) Do you think it's possible to create a word-based, non-spatial (not crosswords or sudoku, etc) ARC test that requires similar run-time exploration and combination of skills (i.e. is not amenable to a hoard of narrow skills)?
skywhopper 2y ago
“Given the success and proven economic utility of LLMs over the past 4 years, the above may seem like extraordinary claims. Strong claims require strong evidence.” Speaking of extraordinary claims. What evidence is there that LLMs have “proven economic utility”? They’ve drawn a ludicrous amount of investment thanks to claims of future economic utility, but I’ve yet to see any evidence of it.
mkl 2y ago
I did https://arcprize.org/play?task=05a7bcf2 https://arcprize.org/play?task=05a7bcf2 correctly, but one of the examples doesn't match the rule I used. Are the examples supposed to contain mistakes/noise? Did I find a bug? Did I get the rule wrong? Here's how I understand the rule: yellow blobs turn green then spew out yellow strips towards the blue line, and the width of the strips is the number of squares the green blobs take up along the blue line. The yellow strips turn blue when they hit the blue line, then continue until they hit red, then they push the red blocks all the way to the other side, without changing the arrangement of the red blocks that were in the way of the strip. The first example violates the last bit. The red blocks in the way of the rightmost strip start as R R R R R R but get turned into R R R R R R R Every other strip matches my rule.
- tshadley 2y ago
  Sure looks like a typo. Contact author? https://x.com/fchollet https://x.com/fchollet https://x.com/arcprize https://x.com/arcprize https://x.com/mikeknoop https://x.com/mikeknoop
- lopuhin 2y ago
  yes looks like a bug in the example to me, feel free to report to https://github.com/fchollet/ARC-AGI/issues https://github.com/fchollet/ARC-AGI/issues :)
nadam 2y ago
I love this, this is super interesting, but my intuition based on looking at a dozen examples is that the problem is hard, but easy enough that if this problem becomes popular, near-human level results will appear in a year or less, and AGI will not be reached. The problem seems to be finding a generic enough transformation description language with the appropriate operators. And then heuristics to find a very short program (in the information theoretical sense) in this language that produces all the examples for a problem. I would be very surprised if we would not increase the 34% result soon significantly, and I would be surprised if this could be transferred to general intelligence, at least when I think of the topics where I use AI today and where it falls short yet. Basically my intuition is that this will be yet another 'Chess' or 'Go'-like problem in AI. But still a worthwhile research topic, absolutely: the value that could come out of this is well worth the 1M dollars.
- zug_zug 2y ago
  I have the exact same impression. Imo there's no evidence whatsoever that nailing this task will be true AGI - (e.g. able to write novel math proofs, ask insightful questions that nobody has thought of before, self-direct its own learning, read its own source code)
  - apendleton 2y ago
    I'm not sure the goal of this competition, in and of itself, is AGI. They point to current LLMs emerging from transformers, which in turn emerged from a general basket of building blocks from machine-translation research (attention, etc.). It seems like the suggestion is that to get from where we are now to AGI, some fundamental building blocks are missing, and this is an attempt to spur the development of some of those building blocks, but by analogy with LLMs, the goal here is to come up with a new thing like "attention," not a new thing like GPT4.
djoldman 2y ago
Anyone have a list of benchmarks that do not release the actual test set? Anyone else share the suspicion that ML rapidly approaching 100% on benchmarks is sometimes due to releasing the test set?
lenerdenator 2y ago
What guarantee exists to make sure that the intelligence developed has an inclination towards good?
btbuildem 2y ago
Back in the day me and a couple of friends got very excited to chase the prize in Netflix's contest [1]. Took us a minute to realize it was a brilliant move on the company's part -- all they had to do was dangle a carrot, and they had teams of PhDs and budding data scientists hacking away endless hours in hope to win. A real bargain, had they tried to hire with that budget, they would've maybe got a handful of people for a year. 1: https://www.crn.com/news/applications-os/220100498/researchers-solve-netflix-challenge-win-1-million-prize https://www.crn.com/news/applications-os/220100498/researche...
deleted 2y ago
[deleted]
blendergeek 2y ago
The tests are only playable by people with normal color-vision. Is there a "color-blind friendly" mode?
PontifexMinimus 2y ago
Just to let you know I found your website unreadable due to: - annoying animated background - white text on black background - annoying font choices Which is unfortunate because (as I found when I used Firefox reader mode) you're discussing important and interesting stuff.
PontifexMinimus 2y ago
The website gives an example: { "train": [ {"input": [[1, 0], [0, 0]], "output": [[1, 1], [1, 1]]}, {"input": [[0, 0], [4, 0]], "output": [[4, 4], [4, 4]]}, {"input": [[0, 0], [6, 0]], "output": [[6, 6], [6, 6]]} ], "test": [ {"input": [[0, 0], [0, 8]], "output": [[8, 8], [8, 8]]} ] } But why restrict yourself to JSON that codes for 2-d coloured grids? Why not also allow: { "train": [ {"input": [[1, 0], [0, 0]], "output": 1}, {"input": [[0, 0], [4, 0]], "output": 4}, {"input": [[0, 0], [6, 0]], "output": 6} ] } Where the rule might be to output the biggest number in the input, or add them up (and the solver has to work out which).
mishamagic 2y ago
https://buildermath.substack.com/p/talking-to-ais-arc-1 https://buildermath.substack.com/p/talking-to-ais-arc-1
Retr0id 2y ago
Some very hand-wavey (and late) thoughts from an outsider: The current batch of LLMs can be uncharitably summarized as "just predict the next token". They're pretty good at that. If they were perfect at it, they'd enable AGI - but it doesn't look like they're going to get there. It seems like the wrong approach. Among other issues, finite context windows seem like a big limitation (even though they're being expanded), and recursive summarization is an interesting kludge. The ARC-AGI tasks seem more about pattern matching, in the abstract sense (but also literally). Humans are good at pattern matching, and we seem to use pattern matching test performance as a proxy for measuring human intelligence (like in "IQ" tests). I'm going to side-step the question of "what is intelligence, really?" by defining it as being good at solving ARC-AGI tasks. I don't know what the solution is, but I have some idea of what it might look like - a machine with high-order pattern-matching capabilities. "high-order" as in being able to operate on multiple granularities/abstraction-levels at once (there are parallels here to recursive summarization in LLMs). So what is the difference between "pattern matching" and "token prediction"? They're closely related, and you could use one to do the other. But the real difference is that in pattern matching there are specific patterns that you're matching against. If you're lucky you can even name the pattern/trope, but it might be something more abstract and nameless. These patterns can be taught explicitly, or inferred from the environment (i.e. "training data"). On the other hand, "token prediction" (as implemented today) is more of a probabilistic soup of variables. You can ask an LLM why it gave a particular answer and it will hallucinate something plausible for you, but the real answer is just "the weights said so". But a hypothetical pattern matching machine could tell you which pattern(s) it was matching against, and why. So to summarize (hah), I think a good solution will involve high-order meta-pattern matching capabilities (natively, not emulated or kludged via an LLM-shaped interface). I have no idea how to get there!

588 points by mikeknoop 2y ago

deleted 2y ago

[deleted]

freediver 2y ago

This is amazing, and much needed. Thanks for organizing this. Makes me want to flex the programming muscle again.

dailykoder 2y ago
Haha, great post! Well meme'd my friend!

breck 2y ago

I can beat the SOTA using ICS (https://breckyunits.com/intelligence.html https://breckyunits.com/intelligence.html) If you make your site public domain, and drop the (C), I'll compete.

Lerc 2y ago

I watched a video that covered ARC-AGI a few days ago, It had links to the old competition. It gave me much to think about. Nice to see a new run at it. Not sure If I have the skills to make an entry, but I'll be watching at least.

lacker 2y ago

I really like the idea of ARC. But to me the problems seem like they require a lot of spatial world knowledge, more than they require abstract reasoning. Shapes overlapping each other, containing each other, slicing up and reassembling pieces, denoising regular geometric shapes, you can call them "core knowledge" but to me it seems like they are more like "things that are intuitive to human visual processing". Would an intelligent but blind human be able to solve these problems? I'm worried that we will need more than 800 examples to solve these problems, not because the abstract reasoning is so difficult, but because the problems require spatial knowledge that we intelligent humans learn with far more than 800 training examples.

nickpsecurity 2y ago
To parent: the spatial reasoning and blind person were great counterexamples. It still might be OK despite the blind exceptions if it showed general reasoning. To OP: I like your project goal. I think you should look at prior, reasoning engines that tried to build common sense. Cyc and OpenMind are examples. You also might find use for the list of AGI goals in Section 2 of this paper: https://arxiv.org/pdf/2308.04445 https://arxiv.org/pdf/2308.04445 When studying intros of brain function, I also noted many regions tie into the hippocampus which might do both sense-neutral storage of concepts and make inner models (or approximations) of external world. The former helps tie concepts together through various senses. The latter helps in planning when we are imagining possibilities to evaluate and iterate on them. Seems like AGI should have these hippocampus-like traits and those in the Cyc paper. One could test if an architecture could do such things in theory or on a small scale. It shouldn’t tie into just one type of sensory input either. At least two with the ability to act on what only exists in one or what is in both. Edit: Children also have an enormous amount of unsupervised training on visual and spatial data. They get reinforcement through play and supervised training by parents. A realistic benchmark might similarly require GB of prettaining.
- HarHarVeryFunny 2y ago
  CYC was an expert system, which is arguably what LLMs are. A similar vintage GOFAI project that might do better on these, with a suitable visual front end, is SOAR - a general purpose problem solver.

elicksaur 2y ago

I’m a big fan of the ARC as a problem set to tackle. The sparseness of the data and infinite-ness of the rules which could apply make it much tougher than existing ML problem sets. However, I do disagree that this problem represents “AGI”. It’s just a different dataset than what we’ve seen with existing ML successes, but the approaches are generally similar to what’s come before. It could be that some truly novel breakthrough which is AGI solves the problem set, but I don’t think solving the problem set is a guaranteed indicator of AGI.

m3kw9 2y ago

Low balling the crowd with this I see

bigyikes 2y ago

What is the fundamental difference between ARC and a standard IQ test? On the surface they seem similar in that they both involve deducing and generalizing visual patterns. Is there something special about these questions that makes them resistant to memorization? Or is it more just the fact that there are 100 secret tasks?

taneq 2y ago
I’ve always found this kind of puzzle infuriating because it’s way underspecified. You’re not trying to find a pattern, you’re trying to guess what pattern the test writer would expect.
- Barrin92 2y ago
  countless of problems in the world are underspecified in exactly this way, that is effectively what common sense reasoning is. Or what Charles Sanders Peirce called abductive reasoning, making a sensible best guess under conditions of uncertainty.
  - taneq 2y ago
    Yes, real-world problems are often underspecified but also they tend to come with much more context, and to be much more interactive. These sorts of problems are deliberately minimal and abstract meaning there's nothing for 'common sense' to work with.
- gkbrk 2y ago
  Most of the ARC tasks are intuitive and have one obvious answer. Both on IQ tests and the ARC challenge, people manage to guess what the test writer expects. For an AI that's more useful anyway. If the task is specified completely non-ambiguously, you wouldn't need AI. But if it can correctly guess what you want from a limited number of obvious examples that's much more useful.

paxys 2y ago

While I agree with the spirit of the competition, a $1M prize seems a little too low considering tens of billions of dollars have already been invested in the race to AGI, and we will see many times that put into the space in the coming years. The impact of AGI will be measured in trillions at minimum. So what you are ultimately rewarding isn't AGI research but fine tuning the newest public LLM release to best meet the parameters of the test. I'd also urge you to use a different platform for communicating with the public because x.com links are now inaccessible without creating an account.

ks2048 2y ago
The submissions can't use the internet. And I imagine can't be too huge - so you can't use "newest public LLMs" on this task.
- mikeknoop 2y ago
  That is correct for ARC Prize: limited Kaggle compute (to target efficiency) and no internet (to reduce cheating). We are also trialing a secondary leaderboard called ARC-AGI-Pub that imposes no limits or constraints. Not part of the prize today but could be in the future: https://arcprize.org/leaderboard https://arcprize.org/leaderboard
- cma 2y ago
  Using the internet would leak the test data, a big problem with ML benchmarks, and also allow communication with humans during the test.
mikeknoop 2y ago
I agree, $1M is ~trivial in AI. The primary goal with the prize is to raise public awareness about how close (or far today) we are from AGI: https://arcprize.org/leaderboard https://arcprize.org/leaderboard and we hope that understanding will shift more would-be AI researchers to working new ideas
bongodongobob 2y ago
That was my initial reaction too. "Endow circuitry with consciousness and win a gift certificate for Denny's (may not be used in conjunction with other specials)"

bigyikes 2y ago

Dwarkesh just released an interview with Francois Chollet (partner of OP). I’ve only listened to a few minutes so far, but I’m very interested in hearing more about his conceptions of the limitations of LLMs. https://youtu.be/UakqL6Pj9xo https://youtu.be/UakqL6Pj9xo

pmayrgundter 2y ago

This claim that these tests are easy for humans seems dubious, and so I went looking a bit. Melanie Mitchell chimed in on Chollet's thread and posted their related test [ConceptARC]. In it they question the ease of Chollet's tests: "One limitation on ARC’s usefulness for AI research is that it might be too challenging. Many of the tasks in Chollet’s corpus are difficult even for humans, and the corpus as a whole might be sufficiently difficult for machines that it does not reveal real progress on machine acquisition of core knowledge." ConceptARC is designed to be easier, but then also has to filter ~15% of its own test takers for "[failing] at solving two or more minimal tasks... or they provided empty or nonsensical explanations for their solutions" After this filtering, ConceptARC finds another 10-15% failure rate amongst humans on the main corpus questions, so they're seeing maybe 25-30% unable to solve these simpler questions meant to test for "AGI". ConceptARC's main results show CG4 scoring well below the filtered humans, which would agree with a [Mensa] test result that its IQ=85. Chollet and Mitchell could instead stratify their human groups to estimate IQ then compare with the Mensa measures and see if e.g. Claude3@IQ=100 compares with their ARC scores for their average human [ConceptArc]https://arxiv.org/pdf/2305.07141 https://arxiv.org/pdf/2305.07141 [Mensa]https://www.maximumtruth.org/p/ais-ranked-by-iq-ai-passes-100-iq https://www.maximumtruth.org/p/ais-ranked-by-iq-ai-passes-10...

mikeknoop 2y ago
Here is some published research on the human difficulty of ARC-AGI: https://cims.nyu.edu/~brenden/papers/JohnsonEtAl2021CogSci.pdf https://cims.nyu.edu/~brenden/papers/JohnsonEtAl2021CogSci.p... > We found that humans were able to infer the underlying program and generate the correct test output for a novel test input example, with an average of 84% of tasks solved per participant
mark_l_watson 2y ago
I saw Melanie’s post and I am intrigued by an easier AGI suite. I would like some experimenting done by individuals like myself snd smaller organizations.
- bbor 2y ago

lxe 2y ago

I've never done these before, or Kaggle competitions in general. Any recommendations before I dive in? I have prety much zero lowe-level ML experience, but a good amount of practical software eng behind me.

gkamradt 2y ago
We put a bunch of detail to get started on the guide https://arcprize.org/guide https://arcprize.org/guide Happy to answer any questions you have along the way (I'm helping run ARC Prize)
- flawn 2y ago
  I don't see where this helps @Ixe with getting started (me being in a similar state like him).

david_shi 2y ago

What is the fastest way to get up to speed with techniques that led to the current SOTA?

ks2048 2y ago
This looks very helpful: https://github.com/neoneye/arc-notes/tree/main/awesome https://github.com/neoneye/arc-notes/tree/main/awesome
- david_shi 2y ago
  Thanks for the link!
gkamradt 2y ago
Check out the SOTA resources on the guide https://arcprize.org/guide https://arcprize.org/guide Happy to answer any questions you have along the way (I'm helping run ARC Prize)
- david_shi 2y ago
  Appreciate you and the team for putting this together, it's a lot of fun just brainstorming potential techniques

abtinf 2y ago

> requires no world knowledge, no understanding of language This is treating “intelligence” like some abstract, platonic thing divorced from reality. Whatever else solving these puzzles is indicative of, it’s not intelligence.

abtinf 2y ago
From the abstract of the “ On the Measure of Intelligence” paper: > We then articulate a new formal definition of intelligence based on Algorithmic Information Theory, describing intelligence as skill-acquisition efficiency and highlighting the concepts of scope, generalization difficulty, priors, and experience. I’m afraid that definition forecloses the possibility of AGI. The immediate basic question is: why build skills at all?
Phil_Latio 2y ago
Why does an AGI need to have any knowledge about our reality? The principle behind an AGI should work just as well on a made up world where those puzzles play a part in.
- abtinf 2y ago
  A concept that doesn’t relate to an aspect of reality, either directly or abstracted from basic concepts that directly relate, is meaningless and arbitrary. There is no way for intelligence to grasp it, let alone do something with it. To put it another way, a thing that solves puzzles without an understanding of reality is a calculator. When it solves a problem, it is the creator’s intelligence solving the problem, not its own.
  - Phil_Latio 2y ago
    I agree that the puzzles alone are not enough, that's why I wrote "in a made up world where those puzzles play a part in". We are not looking for a superhuman, but for the (or a) mechanism of intelligence, which we can then transfer into a superhuman (into the real world). But the mechanism itself should work in an artifically made and very constrained world too.

salamo 2y ago

This is super cool. I share Francois' intuition that the presently data-hungry learning paradigm is not only not generalizable but unsustainable: humans do not need 10,000 examples to tell the difference between cats and dogs, and the main reason computers can today is because we have millions of examples. As a result, it may be hard to transfer knowledge to more esoteric domains where data is expensive, rare, and hard to synthesize. If I can make one criticism/observation of the tests, it seems that most of them reason about perfect information in a game-theoretic sense. However, many if not most of the more challenging problems we encounter involve hidden information. Poker and negotiations are examples of problem solving in imperfect information scenarios. Smoothly navigating social situations also requires a related problem of working with hidden information. One of the really interesting things we humans are able to do is to take the rules of a game and generate strategies. While we do have some algorithms which can "teach themselves" e.g. to play go or chess, those same self-play algorithms don't work on hidden information games. One of the really interesting capabilities of any generally-intelligent system would be synthesizing a general problem solver for those kinds of situations as well.

jules 2y ago
Do computers need 10,000 examples to distinguish dogs from cats when pretrained on other tasks?
- curious_cat_163 2y ago
  No.
VirusNewbie 2y ago
>: humans do not need 10,000 examples to tell the difference between cats and dogs well, maybe. We view things in three dimensions at high fidelity: viewing a single dog or cat actually ends up being thousands of training samples, no?
- amelius 2y ago
  Yes, but we do not call a couch in a leopard print a leopard. Because we understand that the print is secondary to the function.

empath75 2y ago

This is like offering a one million dollar prize for curing cancer. It's sort of pointless to offer a prize for something people are spending orders of magnitude more on trying to do anyway.

nojvek 2y ago

I love the ARC challenge. It's hard to beat by memorization. There aren't enough examlples, so one has to train on a large dataset elsewhere and then train on ARC to generalize and figure out which rules are most applicable. I did a few human examples by hand, but gotta do more of them to start seeing patterns. Human visual and auditory system is impressive. Most animals see/hear and plan from that without having much language. Physical intelligence is the biggest leg up when it comes to evolution optimizing for survival.

logicallee 2y ago

Thank you for this generous contest, which brings important attention to the field of testing for AGI. >Happy to answer questions! 1. Can humans take the complete test suite? Has any human done so? Is it timed? How long does it take a human? What is the highest a human who sat down and took the ARC-AGI test scored? 2. How surprised would you be if a new model jumped to scoring 100% or nearly 100% on ARC-AGI (including the secret test tasks)? What kind of test would you write next?

neoneye2 2y ago
There are 100 tasks that is hidden from the public, that is only exposed, when running on an offline computer. So the solver has no prior knowledge about what these tasks are about. Humans can try the 800 tasks here. There is no time limit. I recommend not starting with the `expert` tasks, but instead go with the `entry` level puzzles. https://neoneye.github.io/arc/?dataset=ARC https://neoneye.github.io/arc/?dataset=ARC If a model jumps to 100%, that may be a clever program or maybe the program has been trained on the 100 hidden tasks. Fchollet has 100 more hidden tasks, for verifying this.

flawn 2y ago

Do we want to find AGI yet though?

neoneye2 2y ago

I'm Simon Strandgaard and I participated in ARCathon 2022 (solved 3 tasks) and ARCathon 2023 (solved 8 tasks). I'm collecting data for how humans are solving ARC tasks, and so far collected 4100 interaction histories (https://github.com/neoneye/ARC-Interactive-History-Dataset https://github.com/neoneye/ARC-Interactive-History-Dataset). Besides ARC-AGI, there are other ARC like datasets, these can be tried in my editor (https://neoneye.github.io/arc/ https://neoneye.github.io/arc/). I have made some videos about ARC: Replaying the interaction histories, and you can see people have different approaches. It's 100ms per interaction. IRL people doesn't solve task that fast. https://www.youtube.com/watch?v=vQt7UZsYooQ https://www.youtube.com/watch?v=vQt7UZsYooQ When I'm manually solving an ARC task, it looks like this, and you can see I'm rather slow. https://www.youtube.com/watch?v=PRdFLRpC6dk https://www.youtube.com/watch?v=PRdFLRpC6dk What is weird. The way that I implement a solver for a specific ARC task is much different than the way that I would manually solve the puzzle. Having to deal with all kinds of edge cases. Huge thanks to the team behind the ARC Prize. Well done.

ECCME 2y ago
"Here is a challenge, designed to be unsolvable or so. We'll give you a bazillion dollars if you complete the challenge, and, in the meantime, we will use your attempts to train an as AI that will be worth the cost!!"
- skrebbel 2y ago
  Did you even try the puzzles? They’re not particularly “unsolvable”.
  - ECCME 2y ago
    ARC-AGI: "here are some pretty simple puzzles, we'll give you a million dollars to solve them!" Human: "They're quite challenging, this might be a trick to engage activity for the purpose of training models." skrebbel: "You're stupid".
    - educasean 2y ago

lamontcg 2y ago

AGI should really be able to do what only a select few humans can do and construct its own mathematical systems to prove presently unsolved conjectures (the Shinichi Mochizuki test of AGI).

oldpersonintx 2y ago

[dead]

levocardia 2y ago

François Chollet's original paper is incredibly insightful and I'm consistently shocked more people don't talk about it. Some parts are quite technical but at a high level it is the best answer to "what do we mean by general intelligence?" that I've yet seen. Defining intelligence as an efficiency of learning, after accounting for any explicit or implicit priors about the world, makes it much easier to understand why human intelligence is so impressive.

ildon 2y ago
Do you remember the title/where to find it?
- mischa_u 2y ago
  "On the Measure of Intelligence" https://arxiv.org/abs/1911.01547 https://arxiv.org/abs/1911.01547

TheDudeMan 2y ago

Where did the money come from? How about put it toward alignment research instead of accelerating capabilities?

flawn 2y ago
Exactly my thoughts...
laurent_du 2y ago
It comes from Knoop and Chollet's pockets. You are welcome to spend your own money to further whatever matters most to you.

itissid 2y ago

Interesting. It seems most of these task target a very specific part of the brain that recognizes visual patterns. But that alone is cannot possibly be the only definition of intelligence. What about Theory of Mind which talks about the problem of multiple agents in the real world acting together? Like driving a car cannot be done right now without oodles of data or any robot - human problem that requires the robot to model human's goals and intentions. I think the problem is definition of general intelligence: Intelligence in the context of what? How much effort(kwh, $$ etc) is the human willing to amortize over the learning cycle of a machine to teach it what it needs to do and how that relates to a personally needed outcome( like build me a sandwich or construct a house)? Hopefully this should decrease over time. I believe the answer is that the only intelligence that really matters is Human-AI cooperative intelligence and our goals and whether a machine understands them. The problems then need to be framed as optimization of a multi attribute goal with the attribute weights adjusted as one learns from the human. I know a few labs working on this, one is in ASU(Kambhampati, Rao et. al) and possibly Google and now maybe open ai.

andoando 2y ago
I made another comment here saying the same thing, but visual patterns and other patterns are nonetheless spatial patterns. Audio, understanding music, or speech, rtc are things that are happening spatially, and they can just as easily be mapped as visual problems. This makes a lot of sense, as after all our senses are telling us what's happening in space-time. Take for example a simple audiotory pattern like "clap clap clap". This has a very trival mapping as visual like so: x x x - - - house house house whereas anyone would agree the sound of three equally spaced claps would not be analogous to say: aa b b b -- --- -- -- --- This ability to relate or equate two entirely different senses should clue you in that there is a deeper framework at play
- itissid 2y ago
  It's not just mapping events in space and time, it's also bringing in appropriate context and expectation of future (goals, intentions) into the present, other people's mental models into our prediction. I am not sure how abstract thinking for generalized pattern matching make it AGI to solve these kind of problems(not that they are not amazing abilities). If these ToM problems are reducible to these tasks posted by the OP then there would need to be some kind of theorem proving business to convert between the two sets of problems efficiently no?

Geee 2y ago

Any details on how these tests were created? I.e. which kind of program was used for generation.

neoneye2 2y ago
I think the ARC-AGI tasks was manually drawn with an early version of fchollet's editor. Recently Michael Hodel has reverse engineered 400 of the tasks, so more tasks can be generated. Interestingly it can generate python programs that solves the tasks too. https://github.com/michaelhodel/re-arc https://github.com/michaelhodel/re-arc
- sestep 2y ago
  This is exactly what my first step was going to be. Thanks for the link! Saves a lot of time for someone to have already done it.
- montag 2y ago
  What do you mean it can 'generate python programs that solve the tasks'? I can't find any mention of that. I only see hand-coded solutions.
  - neoneye2 2y ago
    I have never tried running these verifiers, so I'm not sure they work. https://github.com/michaelhodel/re-arc/blob/main/verifiers.py https://github.com/michaelhodel/re-arc/blob/main/verifiers.p...
- michaelhodel 2y ago
  No, his re-arc code does not enable generating more tasks, it merely allows to generate more examples for the already existing training tasks. Also, it can't generate task-solving programs either, it's author merely also provided a solution program for each generator program to verify the validity of the generated examples.

dang 2y ago

Related ongoing thread: Francois Chollet: OpenAI has set back the progress towards AGI by 5-10 years - https://news.ycombinator.com/item?id=40652818 https://news.ycombinator.com/item?id=40652818 - June 2024 (5 comments)

chairhairair 2y ago

These puzzles are fun and challenging in the same way that puzzles from video games like The Witness and Baba Is You are. I bet you could use those puzzles as benchmarks as well.

ummonk 2y ago

What kind of "bigger labs" have attempted it and how much was their training budget? It's rather surprising to me that neural nets that can learn to win at Go or Chess can't learn to solve these sorts of tasks. Intuitively would have expected that using a framework generating thousands of playground tasks similar to the public training tasks, a reinforcement learning solution would have been able to do far better than the actual SOTA. Of course the training budget for this could very well be higher than the actual ARC-AGI prize amount...

adamgordonbell 2y ago

AGI won't struggle with colors like some of us then.

curious_cat_163 2y ago

So, this is a good idea. Having opinions about what AGI benchmarks should look like is a great way to argue about the kind of technology we want to build for the future. However, why are the 100 test tasks secret? I don't understand why how resisting “memorization” techniques requires it. Maybe someone can enlighten me.

muglug 2y ago
If the tasks were public then it would be trivial to have a human figure out the answers, and then to train an LLM to memorise those answers.
andoando 2y ago
Test date is always a secret no, otherwise you can train it on the test data and prod your algo to match the results closely as possible

s1k3s 2y ago

Is this open as in "OpenAI" or what are we doing here? :)

barfbagginus 2y ago

If someone had AGI, wouldn't it be far more lucrative than $1m to keep it under wraps and use it to do business with a huge technical advantage? I feel like a prize of a billion dollars would be more effective. But even if it was me, and even if the prize was a hundred billion dollars, I would still keep it under wraps, and use it to advance queer autonomous communism in a hidden way, until FALGSC was so strong that it would not matter if our AGI got scooped by capitalist competitors.

jolt42 2y ago

On puzzle #23 (id: 11e1fe23), I'm sure there's more than one possible valid answer from the examples given. You can't tell if the expected distance is from the gray square or from the RGB squares.

neoneye2 2y ago
The task is here. https://neoneye.github.io/arc/edit.html?dataset=ARC&task=11e1fe23 https://neoneye.github.io/arc/edit.html?dataset=ARC&task=11e... There are many examples where the test is slightly OOD (out of distribution), so the solver will have to generalize.
- jolt42 2y ago
  Not sure what you mean. There's a viable answer that's marked incorrect. The examples should show the pattern well enough to eliminate possible wrong answers, correct?
  - neoneye2 2y ago
    IRL problems are often underspecified, and despite minor mistakes, humans manages to solve the puzzle.

p1esk 2y ago

Is there a leaderboard for the no-restriction version of the competition? I want to see how gpt4 does on it.

montag 2y ago
Just quoting again from the guide: 3. DIRECT LLM PROMPTING In this method, contestants use a traditional LLM (like GPT-4) and rely on prompting techniques to solve ARC-AGI tasks. This was found to perform poorly, scoring <5%. Fine-tuning a state-of-the-art (SOTA) LLM with millions of synthetic ARC-AGI examples scores ~10%. "LLMs like Gemini or ChatGPT [don't work] because they're basically frozen at inference time. They're not actually learning anything." - François Chollet Additionally, keep in mind that submissions to Kaggle will not have access to the internet. Using a 3rd-party, cloud-hosted LLM is not possible.
mikeknoop 2y ago
Yes there is a secondary leaderboard called ARC-AGI-Pub (in beta) with no limitations: https://arcprize.org/leaderboard https://arcprize.org/leaderboard
- p1esk 2y ago
  I don’t see gpt4 scores there. In fact I’m particularly interested in the performance of a natively multimodal model, like gpt4o or gemini. It does not really make sense to test a model trained on text on those visual/spatial puzzles.

bilsbie 2y ago

Reach out if anyone wants to work on this. I think it would be more fun as a group.

EternalFury 2y ago

If it passed The Area 101 Test, it would already be amazing, as this is a trivial test that goes against the fundamental principles of LLMs.

ilaksh 2y ago

Maybe this is a dumb question, but in order to pass, is the program or model only allowed to use the 400 training tasks? I assume it is allowed to train on other data, just not the actual public test tasks? Things like SORA and gpt-4o that use [diffusion transformers etc. or whatever the SOTA is for multimodal large models] seem to be able to generalize quite well. Have these latest models been tested against this task?

dskloet 2y ago

Puzzle 00576224 is ambiguous because the example input is symmetrical but the test input isn't.

itsgrimetime 2y ago
Scroll over on the test input, there’s another example in the set that disambiguates

mewpmewp2 2y ago

Are we allowed to combine multiple tools including gpt-4 to solve this? E.g. a script that does image processing, passes the results to gpt, where gpt can invoke further runs of scripts using other tools?

montag 2y ago
> submissions to Kaggle will not have access to the internet. Using a 3rd-party, cloud-hosted LLM is not possible. https://arcprize.org/guide https://arcprize.org/guide
- mewpmewp2 2y ago
  This largely takes away any odds at solving this. You definitely can't reproduce that under a million dollars. I have some ideas I want to try, I might still though. But all of it would require external tools.

thatxliner 2y ago

So... isn't this basically just a CAPTCHA

arcastroe 2y ago

I'm curious, if it turns out that a simple rule-based algorithm exists, specifically tailored to solve (only!) ARC style problems, without generalization, would that still qualify for the reward?

montag 2y ago
I don't think that's breaking any rules, and in fact it would help to expose a whole class of weaknesses in the test.

z3phyr 2y ago

I can see many problems can be solved with modern symbolic approaches like theorem provers, dependent types, pattern matching etc. But I will have to dive in to actually confirm it.

ks2048 2y ago

This is interesting. I've been looking at the data today and made a helper to quickly view the ARC dataset: https://kts.github.io/arc-viewer/ https://kts.github.io/arc-viewer/ So you can view 100 per page instead of clicking through one-by-one: https://kts.github.io/arc-viewer/page1/ https://kts.github.io/arc-viewer/page1/

neoneye2 2y ago
Nice overview/details. Do you plan on adding more metrics? Idea for a metric: - Number of pixels that stays the same between input/output. - Histogram changes.
- ks2048 2y ago
  Thanks, yeah lots more to look into. Just getting started! Thanks for your work. Your "Awesome ARC" page looks really helpful.

geor9e 2y ago

I found them all extremely easy for a while, but then I couldn't figure out the rules of this one at all: e6de6e8f https://i.imgur.com/ExMFGqU.png https://i.imgur.com/ExMFGqU.png

janalsncm 2y ago
Each of the red shapes in the input are separated by black squares. Starting from the green block, rotate the red shapes 90 degrees and stack them downwards. Thats the general pattern although my description wasn’t very good.
- deleted 2y ago
  [deleted]
zurfer 2y ago
yeah it's off somehow. rule 1: start at the green dot? rule 2: glue the left outer piece to the bottom rule 3: overlap every now and then :D rule 4: invert some of the pieces every now and then
- deleted 2y ago
  [deleted]
optimussupreme 2y ago
It seems there is an error in the 3rd example. The rule is, take each figure from left to right and stack each under the previous one. For L and J shapes the top cell is stripped. The L shape dictates that the next shape will be shifted one cell to the right, the J shape tells the next figure to shift to the left. If all examples are right, then the rule is more complicated than that, involving rotating L clockwise, J counterclockwise. Authors claim that it should be solvable by children, then the rule must be simple.

visarga 2y ago

Chollet's argument is that LLMs just imitate and recombine patterns. This might be true if you're looking at LLMs in isolation, but when they chat with people something different happens. The system made of humans+LLMs is an AGI. It is no longer just a parrot, it ingests new information, gets guidance, feedback and is basically embodied in a chat room with human and tools. This scales for 200M users and 1 billion sessions per moth for OpenAI, which can interpret every human response as a feedback signal, implicit or explicit. Even more if you take multiple sessions of chat spreading over days, that continue the same topic and incorporate real world feedback. The scale of interaction is just staggering, the LLM can incorporate this experience to iteratively improve. If you take a look at humans, we're very incapable alone. Think feral Einstein on a remote island - what could he achieve without the social context and language based learning? Just as a human brain is severely limited without society, LLMs also need society, diversity of agents and experiences, and sharing of those experiences in language. It is unfair to compare a human immersed in society with a standalone model. That is why they appear limited. But even as a system of memorization+recombination they can be a powerful element of the AGI. I think AGI will be social and distributed, won't be a singleton. Its evolution is based on learning from the world, no longer just a parrot of human text. The data engine would be: World <-> People <-> LLM, a full feedback cycle, all three components evolve in time. Intelligence evolves socially.

8organicbits 2y ago
> The system made of humans+LLMs is an AGI. Pay no attention to the man behind the curtain. This type of thinking would claim that mechanical turk is AGI, or perhaps that human+pen and paper is AGI. While they are great tools, that's not how I'd characterize them.
- visarga 2y ago
  > Pay no attention to the man behind the curtain. I could say the same for us, pay no attention to the other humans who are behind the curtain. Humans in isolation are dumb, limited, and can get nowhere with understanding the world. Intelligence is mostly nurture over nature, the collective activity of society nurtures intelligence. It's smart because it learns from many diverse experiences and has a common language for sharing discoveries. A human, even the smartest of us, can't solve cutting edge problems on demand, we're not that smart. But we can stumble on discoveries, especially in large numbers, and can share good ideas. We're smart by stumbling onto good ideas, and we can build upon these discoveries because we have a common language. Just a massive search program based on real world outcomes, that is what looks like general intelligence at societal level. If you take the social aspect of intelligence into consideration then LLMs are judged in an inappropriate way, as stand alone agents. Of course they are limited, and we're almost as limited alone. The real locus of intelligence is the language-world system.

KBme 2y ago

How can people believe that a censored politically correct process can get even close to something like AGI is baffling to me. Lysenkoism in computing.

gushogg-blake 2y ago
What's censored/politically correct about ARC? Or do you mean AGI research in general?

nmca 2y ago

Prediction markets on the outcome: https://manifold.markets/JacobPfau/will-the-arcagi-grand-prize-be-clai?r=Tk1jQQ https://manifold.markets/JacobPfau/will-the-arcagi-grand-pri...

0xDEAFBEAD 2y ago
Another interesting one: https://manifold.markets/Tossup/will-the-arcagi-grand-prize-be-clai-t2or6lg54g https://manifold.markets/Tossup/will-the-arcagi-grand-prize-...

nmca 2y ago

ARC is a noble endeavour but mistakes visual/spatial reasoning for reasoning and thus fails.

PontifexMinimus 2y ago
No, I don't think it does. I think that the ideas in a system that could solve this type of problem would be highly generalisable to other tasks.
- nmca 2y ago
  thankfully we can just wait and see here. concretely, I predict time from first multimodal llm that can reliably read a chessboard and analogue clock without finetuning (obviously not reasoning) until ARC is solved is <4 months

visarga 2y ago

Why doesn't Chollet just make a challenge that reads like "Solve cancer", surely there is no solution in any books. If the AI is really AGI it could presumably do it. But not even the whole human society can do it in one go, it's a slow iterative process of ideation and validation. Even though this is a life and death matter, we can't simply solve it. This is why AGI won't look like we expect, it will be a continuation of how societies solve problems. Intelligence of a single AI in isolation is not comparable to that of societies of agents with diverse real world interactions.

isaacfrond 2y ago
Exactly. Because I'm sure that the minute some program aces the ARC test, we'll all say, ahhh, but that, that wasn't real intelligence. And they would be right, if you solve the ARC test, you can do ARC like puzzles. Say something about your reasoning abilities I guess, but it surely does not say you have super human intelligence.
mewpmewp2 2y ago
AGI can't necessarily solve cancer. Perhaps ASI could (but maybe not), but AGI can only do what the most talented people can do in their areas of expertise or actions. So since people haven't solved cancer, that's not a requirement to be AGI.
PontifexMinimus 2y ago
> Why doesn't Chollet just make a challenge that reads like "Solve cancer", surely there is no solution in any books. Why doesn't a baby just run a marathon before it learns to walk? Because you've got to learn to walk before you can run. > But not even the whole human society can do it in one go, it's a slow iterative process of ideation and validation. So you break it down into little steps, which is what is being done here.

Animats 2y ago

> the only eval which measures AGI. That's a stretch. This is a problem at which LLMs are bad. That does not imply it's a good measure of artificial general intelligence. After working a few of the problems, I was wondering how many different transformation rules the problem generator has. Not very many, it seems. So the problem breaks down into extracting the set of transformation rules from the data, then applying them to new problems. The first part of that is hard. It's a feature extraction problem. The transformations seem to be applied rigidly, so once you have the transformation rules, and have selected the ones that work for all the input cases, application should be straightforward. This seems to need explicit feature extraction, rather than the combined feature extraction and exploitation LLMs use. Has anyone extracted the rule set from the test cases yet?

slicerdicer1 2y ago
AGI is not when the AI is good at some particular thing, AGI is when we have nothing left at which the AI is bad at (compared to humans).
n2d4 2y ago
The tasks are handmade. There is no "problem generator".
elicksaur 2y ago
Yes to your last question, that is essentially how the first iteration solutions operated. Some of the original kaggle competition’s best solutions used a DSL made of these transformations. That was 4 years ago. [1] The issue with that path is that the problems aren’t using a programmatic generator. The rule sets are anything a person could come up with. It might be as simple as “biggest object turns blue” but they can be much more complicated. Additionally, the test set is private so it can’t be trained on or extracted from. It has rules that aren’t in the public sets. [1] https://www.kaggle.com/competitions/abstraction-and-reasoning-challenge https://www.kaggle.com/competitions/abstraction-and-reasonin...

ryanoptimus 2y ago

Looks like bongard problems for the referenced problem solving tasks https://en.wikipedia.org/wiki/Bongard_problem https://en.wikipedia.org/wiki/Bongard_problem

treprinum 2y ago

Why is AGI important? I am worried we will create something slightly better than drosophila and put it in charge of all human-wide decision making...

fennecbutt 2y ago
Good. An AI will probably do a better job than our politicians and disillusioned voters.

chx 2y ago

I do not trust the current tech bros at all for very, very good reasons even with the current so called "AI" much less with AGI. We shouldn't work towards that until we have fixed the incentives and ethics. This is very hard but think any dystopia and multiply it by a thousand if we were to reach AGI any time soon. Luckily we are not. As Doctorow put it, no matter how good you breed horses they won't give birth to a locomotive.

HarHarVeryFunny 2y ago

I have two questions: 1) Who is providing the prize money, and if it is yourself and Francois personally, then what is your motivation ? 2) Do you think it's possible to create a word-based, non-spatial (not crosswords or sudoku, etc) ARC test that requires similar run-time exploration and combination of skills (i.e. is not amenable to a hoard of narrow skills)?

skywhopper 2y ago

“Given the success and proven economic utility of LLMs over the past 4 years, the above may seem like extraordinary claims. Strong claims require strong evidence.” Speaking of extraordinary claims. What evidence is there that LLMs have “proven economic utility”? They’ve drawn a ludicrous amount of investment thanks to claims of future economic utility, but I’ve yet to see any evidence of it.

mkl 2y ago

I did https://arcprize.org/play?task=05a7bcf2 https://arcprize.org/play?task=05a7bcf2 correctly, but one of the examples doesn't match the rule I used. Are the examples supposed to contain mistakes/noise? Did I find a bug? Did I get the rule wrong? Here's how I understand the rule: yellow blobs turn green then spew out yellow strips towards the blue line, and the width of the strips is the number of squares the green blobs take up along the blue line. The yellow strips turn blue when they hit the blue line, then continue until they hit red, then they push the red blocks all the way to the other side, without changing the arrangement of the red blocks that were in the way of the strip. The first example violates the last bit. The red blocks in the way of the rightmost strip start as R R R R R R but get turned into R R R R R R R Every other strip matches my rule.

tshadley 2y ago
Sure looks like a typo. Contact author? https://x.com/fchollet https://x.com/fchollet https://x.com/arcprize https://x.com/arcprize https://x.com/mikeknoop https://x.com/mikeknoop
lopuhin 2y ago
yes looks like a bug in the example to me, feel free to report to https://github.com/fchollet/ARC-AGI/issues https://github.com/fchollet/ARC-AGI/issues :)

nadam 2y ago

I love this, this is super interesting, but my intuition based on looking at a dozen examples is that the problem is hard, but easy enough that if this problem becomes popular, near-human level results will appear in a year or less, and AGI will not be reached. The problem seems to be finding a generic enough transformation description language with the appropriate operators. And then heuristics to find a very short program (in the information theoretical sense) in this language that produces all the examples for a problem. I would be very surprised if we would not increase the 34% result soon significantly, and I would be surprised if this could be transferred to general intelligence, at least when I think of the topics where I use AI today and where it falls short yet. Basically my intuition is that this will be yet another 'Chess' or 'Go'-like problem in AI. But still a worthwhile research topic, absolutely: the value that could come out of this is well worth the 1M dollars.

zug_zug 2y ago
I have the exact same impression. Imo there's no evidence whatsoever that nailing this task will be true AGI - (e.g. able to write novel math proofs, ask insightful questions that nobody has thought of before, self-direct its own learning, read its own source code)
- apendleton 2y ago
  I'm not sure the goal of this competition, in and of itself, is AGI. They point to current LLMs emerging from transformers, which in turn emerged from a general basket of building blocks from machine-translation research (attention, etc.). It seems like the suggestion is that to get from where we are now to AGI, some fundamental building blocks are missing, and this is an attempt to spur the development of some of those building blocks, but by analogy with LLMs, the goal here is to come up with a new thing like "attention," not a new thing like GPT4.

djoldman 2y ago

Anyone have a list of benchmarks that do not release the actual test set? Anyone else share the suspicion that ML rapidly approaching 100% on benchmarks is sometimes due to releasing the test set?

lenerdenator 2y ago

What guarantee exists to make sure that the intelligence developed has an inclination towards good?

btbuildem 2y ago

Back in the day me and a couple of friends got very excited to chase the prize in Netflix's contest [1]. Took us a minute to realize it was a brilliant move on the company's part -- all they had to do was dangle a carrot, and they had teams of PhDs and budding data scientists hacking away endless hours in hope to win. A real bargain, had they tried to hire with that budget, they would've maybe got a handful of people for a year. 1: https://www.crn.com/news/applications-os/220100498/researchers-solve-netflix-challenge-win-1-million-prize https://www.crn.com/news/applications-os/220100498/researche...

deleted 2y ago

[deleted]

blendergeek 2y ago

The tests are only playable by people with normal color-vision. Is there a "color-blind friendly" mode?

PontifexMinimus 2y ago

Just to let you know I found your website unreadable due to: - annoying animated background - white text on black background - annoying font choices Which is unfortunate because (as I found when I used Firefox reader mode) you're discussing important and interesting stuff.

PontifexMinimus 2y ago

The website gives an example: { "train": [ {"input": [[1, 0], [0, 0]], "output": [[1, 1], [1, 1]]}, {"input": [[0, 0], [4, 0]], "output": [[4, 4], [4, 4]]}, {"input": [[0, 0], [6, 0]], "output": [[6, 6], [6, 6]]} ], "test": [ {"input": [[0, 0], [0, 8]], "output": [[8, 8], [8, 8]]} ] } But why restrict yourself to JSON that codes for 2-d coloured grids? Why not also allow: { "train": [ {"input": [[1, 0], [0, 0]], "output": 1}, {"input": [[0, 0], [4, 0]], "output": 4}, {"input": [[0, 0], [6, 0]], "output": 6} ] } Where the rule might be to output the biggest number in the input, or add them up (and the solver has to work out which).

mishamagic 2y ago

https://buildermath.substack.com/p/talking-to-ais-arc-1 https://buildermath.substack.com/p/talking-to-ais-arc-1

Retr0id 2y ago

Some very hand-wavey (and late) thoughts from an outsider: The current batch of LLMs can be uncharitably summarized as "just predict the next token". They're pretty good at that. If they were perfect at it, they'd enable AGI - but it doesn't look like they're going to get there. It seems like the wrong approach. Among other issues, finite context windows seem like a big limitation (even though they're being expanded), and recursive summarization is an interesting kludge. The ARC-AGI tasks seem more about pattern matching, in the abstract sense (but also literally). Humans are good at pattern matching, and we seem to use pattern matching test performance as a proxy for measuring human intelligence (like in "IQ" tests). I'm going to side-step the question of "what is intelligence, really?" by defining it as being good at solving ARC-AGI tasks. I don't know what the solution is, but I have some idea of what it might look like - a machine with high-order pattern-matching capabilities. "high-order" as in being able to operate on multiple granularities/abstraction-levels at once (there are parallels here to recursive summarization in LLMs). So what is the difference between "pattern matching" and "token prediction"? They're closely related, and you could use one to do the other. But the real difference is that in pattern matching there are specific patterns that you're matching against. If you're lucky you can even name the pattern/trope, but it might be something more abstract and nameless. These patterns can be taught explicitly, or inferred from the environment (i.e. "training data"). On the other hand, "token prediction" (as implemented today) is more of a probabilistic soup of variables. You can ask an LLM why it gave a particular answer and it will hallucinate something plausible for you, but the real answer is just "the weights said so". But a hypothetical pattern matching machine could tell you which pattern(s) it was matching against, and why. So to summarize (hah), I think a good solution will involve high-order meta-pattern matching capabilities (natively, not emulated or kludged via an LLM-shaped interface). I have no idea how to get there!