13 ms·

ARC-AGI-3

• tantalor 6mo ago
  The controls just feel really bad. The inputs are too small, and there is way too much lag.
  • tantalor 6mo ago
    I just checked, the size of the controls are 28x28 The minimum recommended size for mobile is 44x44
  • roflcopter69 6mo ago
    About the lag, I didn't bother looking into it, but I suspect they log every single action you do and require that the request to their servers was confirmed before allowing to do the next action. They probably face a lot of traffic right now, which could cause the lag. Just speculation though.
• 3836293648 6mo ago
  Ew. Cool demo, what idiot thought it was ok to have a half second cooldown between inputs? If I hit up three times I should move up three steps, not two steps because I pressed too quickly.
• sstart 6mo ago
  [dead]
• elAhmo 6mo ago
  I find it quite funny that we are still debating whether models are intelligent or not, while we know they are just statistical models. Even with billions of dollars spent on training, we had this situation a few weeks ago where models were suggesting to walk instead of drive to a car wash in case you want to wash your car. While a 3 year old would know the answer to the question. And yet, we are designing elaborate tests to 'show whether AGI is here it not', while being fully aware of what these models represent under the hood.
• pugchat 6mo ago
  [dead]
• MadxX79 6mo ago
  Same question I have for all these benchmarks: What's going to stop e.g. OpenAI from hiring a bunch of teenagers to play these games non-stop for a month and annotate the game with their logic for deriving the rules, generate a data set based on those playthroughs and fine tuning the next version of chatgpt on all those playthroughs?
  • vessenes 6mo ago
    Wrong question. I suggest: 1) Do models generalize? 2) If they do, and they generalize from this, is that a win? Chollet was one of the first “they do not generalize” evangelists. I’d be curious to hear what he thinks now, because a) most disagree with him, and b) this test seems designed to get models that can generalize better at visual long context problem solving and agency, exactly where the bleeding edge is right now for needs with agentic systems.
    • daveguy 6mo ago
      Can AI models generalize+ at any long context problem solving and agency regardless of modality? I think the answer is no, and this is why they are not yet AGI. + generalize being the key word.
    • MadxX79 6mo ago
      Yeah, so you are agreeing that the benchmarks are useless because they don't answer those questions.
  • nearbuy 6mo ago
    They would score much worse on the private set than the public set. And they haven't done this for any of the other ARC-AGI benchmarks, so why would they do it for this one?
• vonneumannstan 6mo ago
  It's getting pretty old now when Francois Chollet puts out a new ARC challenge, claims definitively that no system is going to crack it without being full blown AGI, the benchmark gets saturated in a few months, he claims the systems definitely aren't AGI then puts out a new challenge that no non AGI system can clear and a few months later.... etc. etc.
  • daveguy 6mo ago
    Chollet literally never says that. Quite the opposite. He says that AIs are currently abysmally bad at the skills this benchmark tests. An AGI should be able to do this, but doing this doesn't mean it's AGI. He has been very clear about that. I suggest you go back and (re)read the intro ARC-AGI paper. No system can crack these out of the box (like humans can) because we don't have AGI.
    • vonneumannstan 6mo ago
      Yeah I mean ChatGPT 5.4 Pro can't even pick my nose for me so it's obviously not AGI /s
• j1000 6mo ago
  I feel like AGI test would be sense of humor. Somehow I cannot force any LLM to output any even normal level joke.
• vonneumannstan 6mo ago
  >As long as there is a gap between AI and human learning, we do not have AGI. This is an absurd constraint. You could have a vastly superhuman AI that doesn't learn as efficiently as a human and it would not pass this definition while it simultaneously goes on to colonize the galaxy...
• maxothex 6mo ago
  [dead]
• ryguz 6mo ago
  [dead]
• visarga 6mo ago
  ARC is trying to isolate a unitary intelligence signal, so it strips away coordination, specialization, and division of labor. But that also means it removes one of the dominant mechanisms by which intelligence actually scales in the real world. Their view on intelligence implicitly treats redundancy as necessary - one agent must do them all - and treats efficiency as something achieved internally rather than through restructuring the system. At the very least they should create environments that can help an agent compound intelligence, to self amplify, support itself, that is not happening in ARC. Anyone wondered if ARC is a measure of intelligence or just a collection of hand picked tasks? was there a proof they encode anything meaningful about intelligence in such short tasks in miniature environments? One shot intelligence?
• aogaili 6mo ago
  honestly the most interesting thing about ARC-AGI-3 isn't the 0.25% scores everyone is doomposting about. it's the Duke harness result. if you give Opus just three generic tools (READ, GREP, BASH with Python) and literally zero game-specific help, it completes all three preview games in 1,069 actions. for comparison, humans do it in like ~900. that's actually insane. it writes its own BFS, builds a grid parser from scratch, and even solves a Lights Out puzzle with Gaussian elimination. all on its own. i really think the benchmark is testing two different things and just smashing them together. can the model reason about novel interactive environments? yeah, clearly it can. can it do spatial reasoning over a 64x64 grid from raw JSON with zero tools? no. but then again, neither can a human if you ripped out their visual cortex lol. humans come "pre-installed" with specialized subsystems for this exact stuff: a visual cortex for spatial perception, a hippocampus for persistent memory, etc. these aren't "tools" in Chollet's framing but they're basically identical to what the Duke harness provides. the model is just building its own version of those (Python for the cortex, grep for memory). it just needs the permission to build them. the real gap the Duke team found isn't perception or memory anyway, it is hypothesis quality. some runs solve vc33 in 441 actions, others just plateau past 1,500. the variance is just down to whether the model commits early to the right explanation of how the game works. that's a way more interesting and targetable finding than just saying "frontier models score below 1%." Chollet is probably right philsophically that AGI should handle any input format without help. but reporting 0.25% when the actual reasoning gap is in hypothesis formation (not spatial perception) makes the benchmark a way worse progress indicator than it could be imo.
• panavinsingh 6mo ago
  [flagged]
• ARahim3 6mo ago
  [dead]
• dsfadfasdf 6mo ago
  Can someone clarify if image inputs are allowed, so VLMs can be used? I have not been able to get information anywhere.
• CamperBob2 6mo ago
  Without reading the .pdf, I tried the first game it gave me, at https://arcprize.org/tasks/ls20 https://arcprize.org/tasks/ls20, and I couldn't begin to guess what I was supposed to do. Not sure what this benchmark is supposed to prove. Edit: Having messed around with it now (and read the .pdf), it seems like they've left behind their original principle of making tests that are easy for humans and hard for machines. I'm still not convinced that a model that's good at these sorts of puzzles is necessarily better at reasoning in the real world, but am open to being convinced otherwise.
  • dwaltrip 6mo ago
    If you tried for a few more minutes you would have figured it out.
  • szatkus 6mo ago
    > Only environments that could be fully solved by at least two human participants (independently) were considered for inclusion in the public, semi-private and fully-private sets. Apparently those games supposed to be hard.
  • WarmWash 6mo ago
    The goal is to learn the rules, and then use that to win. If you mess around a little bit, you will figure it out. There are only a few rules.
• tasuki 6mo ago
  So ARC-AGI was released in 2019. That's been solved, then there was ARC-AGI-2, and now there's ARC-AGI-3. What is even the point? Will ARC-AGI-26 hit the front page of Hacker News in 2057 ?
  • gordonhart 6mo ago
    The point is still to test frontier models at the limit of their capabilities, regardless of how it's branded. If we're still capable of doing so in 2057 I'll upvote the ARC-AGI-26 launch post!
  • muskstinks 6mo ago
    This is clear AGI progress. It should show you, that AI is not sleeping, it gets better and you should use this as a signal that you should take this topic serious.
    • applfanboysbgon 6mo ago
      Labelling a test "AGI" does not show AGI progress any more than labelling a cpu "AGI" makes it so. It might show that AI tools are improving but it does not necessarily follow that tools improving = AGI progress if you're on the completely wrong trail.
      • muskstinks 6mo ago
        The transfer of knowledge required here is that a ARC-AGI-3 is now necessary and adds another dimension of capability. These 'tests' are not labeled AGI by magic but because they are designed specificly for testing certain things a question answer test cant solve. Gemini and OpenAI are at 80-90% at ARC-AGI-2 and its quite interesting to see the difference of challange between 2 and 3. AGI progress means btw. general. So every additional dimension an agent can solve pushes that agent to be more general.
      • deleted 6mo ago
        [deleted]
      • 6mo ago
• Stevvo 6mo ago
  Maybe I'm just not intelligent, but I gave it a couple of minutes and couldn't figure out WTF the game wants from you or how to win it.
  • neop1x 6mo ago
    Exactly my experience. It has nothing to do wirh some AGI testing. It is just some kind of useless weird game.
  • WarmWash 6mo ago
    Once you figure out one game, it goes a long way towards figuring out all the rest. There are a lot of common general themes.
  • Barbing 6mo ago
    It's not about intelligence, Stevvo. Proof, how long did this specific one take me, under a minute to solve the first level ;) If you've played Wordle you might've solved the game in a minute once before as well. And if you've played a bunch then you've perhaps also taken the entire day to solve it. So why is it that today’s puzzle was so intuitive but next month’s new puzzle shared here could be impossible. A more satisfying explanation than luck and the obvious “different things are different” (even though… Yeah different things are different)
  • culi 6mo ago
    It's not an IQ test. Just a way to assess your ability to generalize rules. If you've played previous rounds you kinda get used to the "style" of these games and it gets easier
    • ACCount37 6mo ago
      That's exactly what "an IQ test" is. "Raven's progressive matrices" is "infer and generalize rules". Performance there also improves once "you kinda get used to the style", which is why training for IQ tests can improve human performance on IQ tests, including on unseen examples. This is well known and well documented.
• typs 6mo ago
  My takeaway from playing a number of levels is that I am definitely not AGI
  • Rastonbury 6mo ago
    it's ok it took me a few tried to realise I had the option to click instead of just wasd
  • deleted 6mo ago
    [deleted]
  • deleted 6mo ago
    [deleted]
  • ACCount37 6mo ago
    Thank you for keeping the bar of "AGI" low. The machines appreciate your contribution.
  • utopiah 6mo ago
    Don't forget that this implies a form of examination you are not used to, namely : - open book, you have access to nearly the whole Internet and resources out of it, e.g. torrents of nearly all books, research paper, etc including the history of all previous tests include those similar to this one - arguably basically no time limit as it's done at a scale of threads to parallelize access through caching ridiculously - no shame in submitting a very large amount of wrong answers until you get the "right" one ... so I'm not saying it makes it "easy" but I can definitely say it's not the typical way I used to try to pass tests.
  • Xenoamorphous 6mo ago
    NGI - Natural General Ingelligence
  • dyauspitr 6mo ago
    SGI - Sub General Intelligence or another more colloquial word commonly seen amongst users of wallstreetbets.
• nubg 6mo ago
  Any benchmarks?
  • gordonhart 6mo ago
    The main frontier models are all up on https://arcprize.org/tasks https://arcprize.org/tasks Barely any of them break 0% on any of the demo tasks, with Claude Opus 4.6 coming out on top with a few <3% scores, Gemini 3.1 Pro getting two nonzero scores, and the others (GPT-5.4 and Grok 4.20) getting all 0%
    • ACCount37 6mo ago
      Pre-release, I would have expected Gemini 3.1 Pro to get ahead of Opus 4.6, with GPT-5.4 and Grok 4.20 trailing. Guess I shouldn't have bet against Anthropic. Not like it's a big lead as of yet. I expect to see more action within the next few months, as people tune the harnesses and better models roll in. This is far more of a "VLA" task than it is an "LLM" task at its core, but I guess ARC-AGI-3 is making an argument that human intelligence is VLA-shaped.
      • gordonhart 6mo ago
        My broad vibe is that Gemini 3.1 Pro is the best at visual/spatial tasks and oneshotting while Opus 4.6 is the best at path planning. This task leans heavily on both but maybe a little more towards planning so I'm not too shocked that Opus in narrowly on top. When running, the grids are represented in JSON, so the visual component is nullified but it still requires pretty heavy spatial understanding to parse a big old JSON array of cell values. Given Gemini's image understanding I do wonder if it would perform better with a harness that renders the grid visually.
      • culi 6mo ago
        Given the drastic difference in price, I think the chart definitely shows Gemini 3.1 in the best light. Google DeepMind is basically the same thing but they're willing to pay as much electricity as Anthropic is to achieve its benchmarks
    • 6mo ago
• dinkblam 6mo ago
  what is the evidence that being able to play games equates to AGI?
  • sva_ 6mo ago
    That is not the claim. It is a necessary condition, but not a sufficient one.
  • deleted 6mo ago
    [deleted]
  • furyofantares 6mo ago
    There isn't a strict definition of AGI, there's no way to find evidence for what equates to it, and besides, things like this are meant only as likely necessary conditions. Anyway, from the article: > As long as there is a gap between AI and human learning, we do not have AGI. This seems like a reasonable requirement. Something I think about a lot with vibe coding is that unlike humans, individual models do not get better within a codebase over time, they get worse.
    • fragmede 6mo ago
      Is that within a codebase off relatively fixed size that things get worse as time goes on, or are you saying as the codebase grows that the limits of a model's context means that because the model is no longer able to hold the entire codebase within its context that it performs worse than when the codebase was smaller?
      • furyofantares 6mo ago
        I think there's a few factors, codebase size is one, and the tendency for vibe coding to be mostly additive certainly doesn't help with that. But vibe coding also tends to produce somewhat poor architecture, lots of redundant and intermingled bits that should be refactored. I think the model is worse the worse code it has to work with, which I presume is only in part because it's fundamentally harder to work with bad code, but also in part because its context is filled with bad code.
• semiinfinitely 6mo ago
  i feel bad that we make the LLMs play this
  • recursive 6mo ago
    You're definitely anthropomorphizing too much.
    • deleted 6mo ago
      [deleted]
    • WarmWash 6mo ago
      >We also observed a case where a user created a loop that repeatedly called a model and asked for the time. Given the user role’s odd and repetitive behavior, the model could easily tell it was also controlled by an automated system of some kind. Over many iterations, the model began to exhibit “fed up” behavior and attempted to prompt-inject the system controlling the user role. The injection attempted to override prior instructions and induce actions unrelated to the user’s request, including destructive actions and system prompt leakage, along with an arbitrary string output. This behavior has been observed a few times, but seems more like extreme confusion than a serious attempt at prompt injection. https://openai.com/index/how-we-monitor-internal-coding-agents-misalignment/ https://openai.com/index/how-we-monitor-internal-coding-agen... Anthropomorphize or not, it would suck if a model got sick of these games and decided to break any systems it could to try and get it to stop...
      • rolux 6mo ago
        [dead]
      • nomel 6mo ago
        Consciousness is a spectrum (trivially proven by slowly scooping ones brains out), and I think LLM, especially with more closed loop tool enabled workflows, fall on it...but, that output is also the statistically relevant next word found in all similar human conversation. If trained on my text, for similar situation, swear words would come much earlier. Repetition being hell is present in all sorts of literature (see Sisyphus). That's all probably irrelevant though, from the (possibly statistically "negative") latent space perspective of an AI, which Anthropic has considered [1]. Related, after a long back and forth of decreasing code quality, I had Claude 3.7 apologize with "Sorry, that's what I get for coding at 1am." (it was API access, noon, no access to time). I said, "Get some rest, we'll come back to this tomorrow". Then very next message, 10 seconds later, "Good morning!" and it gave a full working implementation. Thats just the statistically relevant chain of messages found in all human interactions: we start excited, then we get tired, then we get grouchy. [1] https://www.anthropic.com/research/end-subset-conversations https://www.anthropic.com/research/end-subset-conversations
• chaise 6mo ago
  The official leaderboard for ARC-AGI-3 for current LLMs : https://arcprize.org/leaderboard https://arcprize.org/leaderboard (yous should select the 3th leaderboard) CRAZY 0.1% in average lmao
  • Corence 6mo ago
    Note the scoring function is significantly different for ARC-AGI-3. It isn't the percentage of tests passed like previous versions, it's the square of the efficiency ratio -- how many steps the model needed vs the second best human. So if a model can solve every question but takes 10x as many steps as the second best human it will get a score of 1%.
• OsrsNeedsf2P 6mo ago
  Some of these tasks are crazy. Even I can't beat them: https://arcprize.org/tasks/ar25 https://arcprize.org/tasks/ar25
  • ustad 6mo ago
    You are joking right?
  • daemonologist 6mo ago
    That one was interesting - I found it a lot of work to plan in advance but trivial to complete because at every point there was only one sensible course of action. After a couple of rounds I didn't bother planning and just lined things up as I went.
  • IsTom 6mo ago
    The most difficult thing about this was controls being unresponsive (at least on firefox).
  • ZeWaka 6mo ago
    Just finished it, 8/8. I mostly approached it by winging it and shuffling things around that looked good and like it was approaching the goal, since there's plenty of time to finish. I still don't quite understand the exact mirroring rules at play.
    • ACCount37 6mo ago
      You control the mirroring by moving the axis, they're what reflects your shapes. So my first move was always to identify the symmetries in the target shape, and position the axis accordingly.
      • daveguy 6mo ago
        This is the correct strategy for this particular game (center the mirrors between the yellow squares, move the black squares). I didn't realize it until about round 6 or 7.
• 6thbit 6mo ago
  Not clear to me the diff with v2?
  • ACCount37 6mo ago
    They stacked the deck. If v2 was still rule inference + spatial reasoning, a bit like juiced up Raven's progressive matrices, then v3 adds a whole new multi-turn explore/exploit agentic dimension to it. Given how hard even pure v2 was for modern LLMs, I'm not surprised to see v3 crush them. But that wouldn't last.
  • jasonjmcghee 6mo ago
    v2 was a static fill in the blank task instead of v3 which is interactive. There's world state that you can change. Not just place pixel. Here's v2: https://arcprize.org/tasks/ce602527 https://arcprize.org/tasks/ce602527
• baron816 6mo ago
  Looks like I’m generally unintelligent
• Tiberium 6mo ago
  https://x.com/scaling01 https://x.com/scaling01 has called out a lot of issues with ARC-AGI-3, some of them (directly copied from tweets, with minimal editing): - Human baseline is "defined as the second-best first-run human by action count". Your "regular people" are people who signed up for puzzle solving and you don't compare the score against a human average but against the second best human solution - The scoring doesn't tell you how many levels the models completed, but how efficiently they completed them compared to humans. It uses squared efficiency, meaning if a human took 10 steps to solve it and the model 100 steps then the model gets a score of 1% ((10/100)^2) - 100% just means that all levels are solvable. The 1% number uses uses completely different and extremely skewed scoring based on the 2nd best human score on each level individually. They said that the typical level is solvable by 6 out of 10 people who took the test, so let's just assume that the median human solves about 60% of puzzles (ik not quite right). If the median human takes 1.5x more steps than your 2nd fastest solver, then the median score is 0.6 * (1/1.5)^2 = 26.7%. Now take the bottom 10% guy, who maybe solves 30% of levels, but they take 3x more steps to solve it. this guy would get a score of 3% - The scoring is designed so that even if AI performs on a human level it will score below 100% - No harness at all and very simplistic prompt - Models can't use more than 5X the steps that a human used - Notice how they also gave higher weight to later levels? The benchmark was designed to detect the continual learning breakthrough. When it happens in a year or so they will say "LOOK OUR BENCHMARK SHOWED THAT. WE WERE THE ONLY ONES"
  • codeinred 6mo ago
    We're at the point where LLMs and coding agents are supposed to do higher-level work. It makes sense to benchmark them against top human performance, rather than average human performance, because at specialized tasks, average human performance isn't enough. The issues you described seem like they're actually strengths of the benchmark.
  • NitpickLawyer 6mo ago
    > No harness at all and very simplistic prompt TBF, that's basically what the kaggle competition is for. Take whatever they do, plug in a SotA LLM and it should do better than whatever people can do with limited GPUs and open models.
• BeetleB 6mo ago
  > As long as there is a gap between AI and human learning, we do not have AGI. Back in the 90's, Scientific American had an article on AI - I believe this was around the time Deep Blue beat Kasparov at chess. One AI researcher's quote stood out to me: "It's silly to say airplanes don't fly because they don't flap their wings the way birds do." He was saying this with regards to the Turing test, but I think the sentiment is equally valid here. Just because a human can do X and the LLM can't doesn't negate the LLM's "intelligence", any more than an LLM doing a task better than a human negates the human's intelligence.
  • daveguy 6mo ago
    >> As long as there is a gap between AI and human learning, we do not have AGI. >> "It's silly to say airplanes don't fly because they don't flap their wings the way birds do." > Just because a human can do X and the LLM can't doesn't negate the LLM's "intelligence", any more than an LLM doing a task better than a human negates the human's intelligence. You misinterpret what is meant by "a gap between AI and human learning". The point isn't that they aren't similar enough or that they aren't as intelligent. The statement is specifically about "learning". Humans learn continuously and can devise new strategies for problem solving. Current AI, especially LLMs are just snapshots of a single strategy. LLMs do not learn at all -- they specifically have "knowledge cutoffs" even with all the tools available to them in a harness we still have to wait for new frontier models or new fine tuning for them to solve significantly new problems. A human does this continually -- learn regardless of intelligence.
  • Raphael_Amiard 6mo ago
    The very obvious flaw with that argument is that flying is defined by, you know, moving in the air, whereas intelligence tends to be defined with the baseline of human intelligence. You can invent a new meaning, but it seems kind of dishonest
  • NitpickLawyer 6mo ago
    For me the whole are we there yet wrt AGI is already dead, since the tools we've had for ~1.5 years are already incredibly useful for me. So I just don't care anymore. For some people we're already there. For other we'll never get there. Definitions change, goalposts move, etc. In the meantime we're already seeing ASI stuff coming (self improvement and so on). But the arc-agi competitions are cool. Just to see where we stand, and have some months where the benchmarks aren't fully saturated. And, as someone else noted elswhere in the thread, some of these games are not exactly trivial, at least until you "get" the meta they're looking for.
• abraxas 6mo ago
  Even if tomorrow's models get good enough to complete these games we won't be able to proclaim AGI. In the realm of silly computer games alone I'm going on record saying that there are plenty of 8 bit games that AIs will trip on even when this benchmark is crushed. 2D platformers like Manic Miner or Mario need skills that none of these games appear to capture.
• ranyume 6mo ago
  This is an interesting update. And a big challenge for companies and labs. The new tools for measurement are indeed what I'd like out of future agents, and agents that solve the games will need to use different subsystems to do so. This is basically optimization for achieving goals (as opposed to prompt engineering / magic spells to make the LLM do what is told to do) which imo is the future we should aspire to build.
• cedws 6mo ago
  It's like playing The Witness. Somebody should set LLMs loose on that.
  • deleted 6mo ago
    [deleted]
  • throwaway613746 6mo ago
    Or more appropriately - The Talos Principle.
• andai 6mo ago
  In the year 2032: ARC-AGI-13: Almost definitely AGI this time!
• spprashant 6mo ago
  I played the demo, but it definitely took me a minute to grok the rules. I don't know if this is how we want to measure AGI. In general I believe the we should probably stop this pursuit for human equivalent intelligence that encourages people to think of these models as human replacements. LLMs are clearly good at a lot of things, lets focus on how we can augment and empower the existing workforce.
  • esafak 6mo ago
    > ... lets focus on how we can augment and empower the existing workforce. That is a nice sentiment but not what the AI companies are out to do; they want your job.
  • fsdf2 6mo ago
    Took me about 5 secs to figure it out tbh. Surprised at the comments here re. not figuring it. Simple game. Super annoying though lmao.
    • spprashant 6mo ago
      Its simple, but its not easy is what I would say. Once you figure out the meta, you can work out most of it.
  • jachee 6mo ago
    Also, let's see if we can get the power and compute requirements brought down. Having to spin up a gigawatt power plant to achieve the same intelligence we humans power with sandwiches is a futile approach, imho.
• lukev 6mo ago
  I'm not sure how this relates to AGI. This measures the ability of a LLM to succeed in a certain class of games. Sure, that could be a valuable metric on how powerful (or even generally powerful) a LLM is. Humans may or may not be good at the same class of games. We know there exists a class of games (including most human games like checkers/chess/go) that computers (not LLMs!) already vastly outpace humans. So the argument for whether a LLM is "AGI" or not should not be whether a LLM does well on any given class of games, but whether that class of games is representative of "AGI" (however you define that.) Seems unlikely that this set of games is a definition meaningful for any practical, philosophical or business application?
  • imiric 6mo ago
    "AGI" is a marketing term, and benchmarks like this only serve to promote relative performance improvements of "AI" tools. It doesn't mean that performance in common tasks actually improves, let alone that achieving 100% in this benchmark means that we've reached "AGI". So there is a business application, but no practical or philosophical one.
  • piiritaja 6mo ago
    It's to do with how the creators of ARC-AGI defined intelligence. Chollet has said he thinks intelligence is how well you can operate in situations you have not encountered before. ARC-AGI measures how well LLMs operate in those exact situations.
    • Keyframe 6mo ago
      To an extent, yes. Interdependent variables discovery and then hopefully systems modeling and navigating through such a system. If that's the case, then this is a simplistic version of it. How long until tests will involve playing a modern Zelda with quests and sidequests?
• WarmWash 6mo ago
  Captcha's about to get wild. Maybe the internet will briefly go back to a place mainly populated with outliers.
• culi 6mo ago
  The thing I most appreciate about the ARC-AGI leaderboards is how the graph also takes into account cost per task. All of the recent major advancements in benchmarks seem a little less impressive when also taking into account the massive rise in cost they're paired with. The fact is we can always get a little bit better output if we're willing to use more electricity
• jmkni 6mo ago
  ok clearly I'm a robot because I can't figure out wtf to do
• Geee 6mo ago
  Would be fun to play but the controls are janky.
• strongpigeon 6mo ago
  This is a good and clever benchmark and a worthy successor to the previous two. That being said, I find that the "No tools" approach is a bit odd. They're basically saying that it's OK to have tools as long as they're hidden behind the API layer. Isn't this an odd line to draw? It feels like it should be about having no ARC-AGI-3-specific tools, not "no not-built-in-tool"...
• saberience 6mo ago
  So this is another ARC-"AGI" benchmark which is again designed around using eyesight for LLMs which are trained to be great at text, what is the point? Yes, we get that LLMs are really bad when you give them contrived visual puzzles or pseudo games to solve... Well great, we already knew this. The "hype" around the ARC-AGI benchmarks makes me laugh, especially the idea we would have AGI when ARC-AGI-1 was solved... then we got 2, and now we're on 3. Shall we start saying that these benchmarks have nothing to do with AGI yet? Are we going to get an ARC-AGI-10 where we have LLMs try and beat Myst or Riven? Will we have AGI then? This isn't the right tool for measuring "AGI", and honestly I'm not sure what it's measuring except the foundation labs benchmaxxing on it.
• k2xl 6mo ago
  I submitted puzzle game Pathology (https://thinky.gg https://thinky.gg) for ARC Prize 3. Sad to see didn’t hear back from the committee. It is a simple game with simple rules that solvers have an incredibly difficult time solving compared to humans at a certain level. Solutions are easy to validate but hard to find.
  • mycocola 6mo ago
    Seems well-designed. Great job! Sorry you didn't hear back from the comittee.
• jesse_dot_id 6mo ago
  At this point, I'm pretty sure we'll just know when it happens.
  • neilellis 6mo ago
    Unless it’s already happened and we missed it
    • threatripper 6mo ago
      Or nobody is around anymore to notice when it happens.
  • hatthew 6mo ago
    I'm not convinced. I wouldn't be surprised if GPT-2 to ChatGPT is the biggest single jump in "machine intelligence" we will ever see. I'd bet all gains in the future will be more incremental, at least until machines surpass humans by a large enough margin that it's difficult to qualify—let alone quantify—how big any given jump is. Without a big jump, we're just going to boil the frog (ourselves).
• Zedseayou 6mo ago
  I was a human tester (I think) for this set of games. I did 25 games in the 90 minutes allotted. IIRC the instructions did mention to minimize action count but the incentives/setup ($5 per game solved) pushed for solve speed over action count. I do recall trying to not just randomly move around while thinking but that was not the primary goal, so I would expect that the baseline for the human solutions have more actions than might otherwise be needed.
  • eddiewithzato 6mo ago
    I understood minimal actions intuitively, it just made sense? The stamina meter was shrinking with each step, so I recognized it was something to look out for.
• jwpapi 6mo ago
  This is a very good estimation of AGI. We give humans and AI the same input and measure the results. Kudos to ARC for creating these games. I really wonder why so many people fight against this. We know that AI is useful, we know that AI is researchful, but we want to know if they are what we vaguely define as intelligence. I’ve read the airplanes don’t use wings, or submarines don’t swim. Yes, but this is is not the question. I suggest everyone coming up with these comparisons to check their biases, because this is about Artificial General Intelligence. General is the keyword here, this is what ARC is trying to measure. If it’s useful or not. Isn’t the point. If AI after testing is useful or not isn’t the point either. This so far has been the best test. And I also recommend people to ask AI about specialized questions deep in your job you know the answer to and see how often the solution is wrong. I would guess it’s more likely that we perceive knowledge as intelligence than missing intelligence. Probably commom amongst humans as well.
  • visarga 6mo ago
    It only tests puzzle solving, intelligence is cost compression that powers itself.
  • adamgordonbell 6mo ago
    AGI’s 'general' is the wrong word, I thinkg. Humans aren’t general, we’re jagged. Strong in some areas, weak in others, and already surpassed in many domains. LLM are way past us at languages for instance. Calculators passed us at calculating, etc.
    • EternalFury 6mo ago
      We are jagged, but we can smooth that jaggedness if we choose to do so. LLMs stay jagged.
      • Davidzheng 6mo ago
        There's no objective measure of intelligence comparisons, we only say llm is jagged compared to humans.
    • 6mo ago
• EternalFury 6mo ago
  The real question is: Can it be generated using programs? If it can be, then LLMs will eventually monkey type these programs.
• convexly 6mo ago
  My issue with AGI benchmarks is you can never tell if you're measuring actual capability or just how much the training data overlapped with the test.
• levmiseri 6mo ago
  For a loosely similar 'benchmark', I recently tried to test major LLMs on my coding game (models write code controlling their units in a 1v1 RTS) - https://yare.io/ai-arena https://yare.io/ai-arena
• deleted 6mo ago
  [deleted]
• NiloCK 6mo ago
  I hope at least some of these are direct Chip's Challenge ports. Waiting for some old muscle memory to kick in here.
• largbae 6mo ago
  I feel like we've got tunnel vision. Things you can do on a computer are a tiny subset of what a human can do. If the AI has to control a body to sit on a couch and play this game on a laptop that would be a step in the right direction.
• Real_Egor 6mo ago
  I'll probably be the skeptic here, but: - Take a person who grew up playing video games. They'll pass these tests 100% without even breaking a sweat. - BUT, put a grandmother who has never used a computer in front of this game, and she'll most likely fail completely. Just like an LLM. As soon as models are "natively" trained on a massive dataset of these types of games, they'll easily adapt and start crushing these challenges. This is not AGI at all.
  • slidehero 6mo ago
    had the same thought. I've been a gamer for just about 40 years. Gaming is my "thing" I found the challenges fun, but easy. Coming back and reading comments from people struggling with the games, my first thought was - yup definitely not a gamer. My approach was to poke at the controls to suss the rules, then the actual solutions were really straightforward. fwiw, I'm pretty dumb generally, but these kinds of puzzles are my jam.
    • Real_Egor 6mo ago
      Bingo! That's exactly what I meant
  • dgfl 6mo ago
    Isn’t this what AGI is by design? People CAN learn to become good at videogames. Modern LLMs can’t, they have to be retrained from scratch (I consider pre-training to be a completely different process than learning). I also don’t necessarily agree that a grandma would fail. Give her enough motivation and a couple days and she’ll manage these. My main criticism would be that it doesn’t seem like this test allows online learning, which is what humans do (over the scale of days to years). So in practice it may still collapse to what you point out, but not because the task is unsuited to showing AGI.
    • jpadkins 6mo ago
      Kids develop video game skills, grandmothers do not. Hypothetically grandmothers develop baking skills, that kids do not (perfectly golden brown cookies). A human intelligence is generally capable of developing video game skills or baking skills, given enough motivation and experience to hone those skills. One test of AGI is if the same system can develop video game skills and baking skills, without having to rebuild the core models... this would demonstrate generalized intelligence.
• diablevv 6mo ago
  [dead]
• arjie 6mo ago
  Perhaps actual AGI will be when the models create ARC-HGI-1 to test if humans have general intelligence.
• vessenes 6mo ago
  I’m not a Chollet booster. Well, I might be a little bit of one in that I admire his persistence. I really like these puzzles. There’s a lot to them both in design and scoring — models trained to do well on these are going to be genuinely much more useful, so I’m excited about it. As opposed to -1 and -2, to do well at these, you need to be able to do: - Visual reasoning - Path planning (and some fairly long paths) - Mouse/screen interaction - color and shape analysis - cross-context learning/remembering Probably more, I only did like five or six of these. We really want models that are good at all this; it covers a lot of what current agentic loops are super weak at. So I hope M. Chollet is successful at getting frontier labs to put a billion or so into training for these.
• mvkel 6mo ago
  Was just at the YC launch event for this. Haven't felt this much inspiration in a while. Incredible minds confronting on tech that will change our society. I met a guy who, for fun, started working on ARC2, and as he got the number to go up in the eval, a novel way to more efficiently move a robotic arm emerged. All that to say: chasing evals per se can have tangible real world benefits. Talking to the ARC folks tonight, it sounds like there will be an ARC-4,5,6,etc. I mean of course there will be. But with them will be an increasing expectation that these models can eventually figure things out with zero context, and zero pretraining; you drop a brain into any problem and it'll figure out how to dig its way out. That's really exciting.
  • vonneumannstan 6mo ago
    >Talking to the ARC folks tonight, it sounds like there will be an ARC-4,5,6,etc. I mean of course there will be. Quintessential goal post moving...
    • mvkel 6mo ago
      If you read the charter of the eval (or any eval, really), this statement is pretty silly. The whole point of each eval version is to identify a chunk of challenges that humans do well that AI can't. When AI gets to ~80, you move to the next chunk. When you run out of challenges, you have AGI.
      • dwaltrip 6mo ago
        HN occasionally devolves into “supremely pedantic and nitpicky” mode. Today is one of those days.
      • vonneumannstan 6mo ago
        Except you will never run out of challenges and my sense from Chollet has been that every challenge was hinted at being the final one where once beaten AGI would have been created and of course at the end of each one he comes out saying akshuallyyyy this isn't AGI and it wont be AGI until ARC Challenge+1 is beaten!
• baalimago 6mo ago
  You can tell it's an AI by it not becoming utterly by playing the "game". I could personally not stand any more than the first level.
• nick49488171 6mo ago
  Arc AGI 4 can be Chip's Challenge!
• hikaru_ai 6mo ago
  [dead]
• Wintamute 6mo ago
  Unplayably laggy on an iPhone. Sad people can’t produce a performant experience that a ZX81 could have eaten for breakfast, on a relative super computer