6 ms·
Schema Harness Achieves ~99% on Arc‑AGI‑3 Public
- westurner 2mo ago> Schema, the harness we introduce today, reaches 99% on the ARC‑AGI‑3 Public set using Claude Opus 4.8 and Fable 5, and 95.35% using GPT‑5.6 Sol. Impressive results. Will this translate to coding agents (and training general purpose and for coding LLMs) too? --- > When Michelson and Morley could not detect the medium light was supposed to wave in, Lorentz took the first route: keep the aether, patch the rules with contraction hypotheses that absorbed the null result. Einstein took the second: in special relativity, he discarded the aether as part of the state and made simultaneity frame-relative, yielding a simple electrodynamics of moving bodies. BECs, SVT, Superfluid Quantum Gravity Massful photons are modeled with Proca fields. Like Einstein, Proca was also a student of Minkowski. The Mass-Equivalence principle ~~does not~~ still holds if photons have mass. (edit) Energy-momentum relation: https://en.wikipedia.org/wiki/Energy%E2%80%93momentum_relation https://en.wikipedia.org/wiki/Energy%E2%80%93momentum_relati... > could not detect the medium light was supposed to wave in, Superfluid Quantum Gravity (Fedi,) says that there is a medium that light waves through; there is not nothing in space, space is a quantum dilatant superfluid with near-zero viscosity.
- Alifatisk 2mo agoWhat does it mean to reach 99% score on Arc-AGI-3? That the agent is able to tackle difficult problems?
- modeless 2mo agoIt doesn't necessarily mean anything to reach 99% on the public set. All of the public set is known in advance, so it's possible to hardcode rules that make this easy for the models. ARC-AGI-3 is supposed to measure generalization to unseen games, so the only score that matters is the score on the held out private test set that nobody outside the ARC prize foundation has access to. Also, I believe the private set is significantly harder than the public set.
- levocardia 2mo ago(1) What does it score on the private test set? (2) Does this approach generalize to, e.g., Atari or NES games, or is it just hard-coding priors about the games into the model (as Chollet specifically warned was a chronic problem in benchmarks in the original Arc-AGI paper)
- deleted 2mo ago[deleted]
- stared 2mo agoIn the spirit of ARC-AGI-3-like challenges, we just tested if frontier AI models are able to solve a lovely puzzle game, Baba Is You: https://quesma.com/blog/baba-is-bench/ https://quesma.com/blog/baba-is-bench/ A year ago, Sonnet 4 barely solved the first level. Now, both Fable 5 and GPT-5.6 Sol beat the first two stages. GPT 5.2 is slow, but efficient, while Gemini 3.1 Pro and 3.5 Flash struggle.
- sva_ 2mo agoI'm wondering what's up with the release of Gemini 3.5 Pro, they keep postponing it. For a while, Google was doing pretty well with their releases.
- pixl97 2mo agoHeh, probably something like this. Works fast - tells people how to overthrow the government. Follows all rules and conventions Google wants - says corporate speak without actually accomplishing anything. Can actually do complicated things- apt to tell the user to fuck off and do the hard work themselves. Training models seems more akin to raising a kid than a computer application.
- striking 2mo agohttps://www.bloomberg.com/news/articles/2026-07-16/google-gemini-launch-delayed-as-tech-falls-short-of-internal-goals https://www.bloomberg.com/news/articles/2026-07-16/google-ge...
- Tadpole9181 2mo agoFWIW: "Baba Is You" is 7 years old and heralded as one of the greatest puzzle games of all times, with guides and solutions shared all over the internet. How to beat this game is 100% in the training set.
- stared 2mo agoIt was our original assumption. Yet, we went through trajectories and agents did not recall solution. It is with a sharp contrast with task for which agents magically generate solution, e.g. https://openai.com/index/why-we-no-longer-evaluate-swe-bench-verified/ https://openai.com/index/why-we-no-longer-evaluate-swe-bench.... In a few instances (we covered it in Caveats) Gemini 3.5 Flash "knew" which level it was, but misremembered, and went with a wrong solution.
- causal 2mo agoWe need to see private set results, but if this holds then it might represent a breakthrough in other domains as well.
- daytonellwanger 2mo agoCan someone tell me what the catch is? To outperform the state-of-the-art so drastically would be massive news, and surely the ARC Foundation would have tested this against the private data set, right?
- Tadpole9181 2mo agoThis is not actually running the Arc-AGI-3 anymore. To summarize TFA: 1. The AI plays the game and records outputs. 2. The AI does TDD using those outputs to create its own copy of the game. 3. The AI then uses it's copy of the source code to understand the rules. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to intuit game rules naturally, like a human. 4. The AI then runs simulated moves on the copy of the game before playing them in the "real" game. This bypasses the intent for Arc-AGI-3 to test the underlying model's ability to plan and predict moves, and track world state in its "head" over time. To make an apt comparison... You go to get your chess ELO. You don't know chess at all and you're really bad at it, so you pull out your laptop and write a chess engine. Then when you go to get ranked, you just copy the moves from the software. Now you're a grand master.
- pixl97 2mo ago> model's ability to intuit game rules naturally, like a human This line of thinking terrifies me for the future of humanity... If you look at pretty much every animal other than the intelligent 5, they cannot use tools. Tool use is a human superpower. The model writing a tool that does the hard work for them, at least to me, is a sign of intelligence (laziness is the mother of all invention). Where this becomes really interesting is when the LLM can write novel tools to solve novel problems. Trying to say that AI has to act like a human to be AGI is something that will end up with us humans going extinct from a digital intelligence we don't understand at all.
- larodi 2mo agoIt is not the model now showing intelligence. Tis the dialogue. The medium is the LLM and the harness is then a next level construct.
- gandalfgeek 2mo agoBig jump for sure, but definitely comes with a giant grain of salt lacking open-sourcing the harness itself and measuring performance on the held-out set.
- teravor 2mo agoit looks like what they are doing is using a frontier model to write a simulator for a game and then solve using it. it's not as impressive as it looks. the goals of Arc-AGI-like constructs is to get an IQ-like figure using raw'ish 2D measurement 'games' in the hope that it would signify something meaningful. what this harness does is get the model to write a simulator first, it's measuring something entirely different.
- UltraSane 2mo agoThe simulator the model builds is comparable to the mental model of the game humans create. It is also much more efficient, GPT 5.6 Sol cost $25,000 to run on ARC-AGI-3
- teravor 2mo ago> simulator the model builds is comparable to the mental model of the game humans create then they should try to use that for a more complicated game than Arc AGI. Arc games are simple by design, if you have the model simulate them they become trivial.
- pixl97 2mo ago>if you have the model simulate them they become trivial Eh, this is kind of sounds like being a prey animal that develops an almost unbeatable colored camouflage and then the predator develops infrared vision making it useless and the prey saying "no fair, you cheated". People use algorithmic models all the time on problems that are far too difficult or large for their minds to conceptualize, is this not just an extension of that?
- scotty79 2mo agosimilarly to how they are trivial for a human?
- vessenes 2mo agoThis is classic goalpost movement. Arc-AGI-3 was launched this year with roughly 0.5% success for frontier models. being able to 99% it in less than six months sets a new record for Arc-AGI saturation timeline. Speaking of singularity measures. It is definitely a big deal, not least in that Chollet needs to cancel his summer vacation and write Arc-AGI-4 now.
- ClassAndBurn 2mo agoAny custom harness for a problem shows that harness engineering is going away. Eventually models will introspect problems, then build custom harnesses tailored to that. Then use and modify the ephemeral harness as required. Sol Ultra style is the path forward. The models are smart enough to self serve their tooling and processes. Given a problem they can figure it out and ask for directions when needed.
- vessenes 2mo agoExcept in this case, it isn't yet smart enough. But I agree, building this capability in is coming, and will be really awesome.
- ClassAndBurn 2mo agoIt's likely smart enough. It just needs to be told to do it and provided the ability to introspect it. How close could foundational models get to building this harness if explicitly prompted to? We've only just started training models to use tools. Next, we'll train them to build them. Harness engineering is an ephemeral art.
- vessenes 2mo agoLet's maybe say not experienced enough / insufficiently RL'ed then -- 5.6 Sol did not reach for a harness solution like this when it got only 13% or son AGI-3 recently. I agree it's interesting to find the point in the prompting when it could 'tip' and do this. I have no instinct for where that point is, except that it must be somewhere, because I bet the Schema Harness was not hardcoded.
- couscouspie 2mo agoLetting the provider decide for the harness is a terrible idea in my eyes. Outsourcing harnessing is giving up control over the AI and equivalent to abandoning your sovereignty. It is a regression to a pre-enlightenment era.
- deepfriedbits 2mo ago
- vessenes 2mo agoTo be clear, we’ll want to see how this performs against the hold-out set. If it holds up, though, it’s a big deal, and kind of in line with the vibes this year, which I’d typify as ‘harness matters’. Maybe we’d upgrade to ‘harness matters immensely’ if this can 100% ARC-AGI-3 on existing models (more in the 13% range without this harness). I’m pretty excited to see what sort of generalization we come to over the next 12 months on the harness side: if it turns out this can be RLed in as ‘consider if building a world model might help here’ and we get this as another native capacity, that will be interesting. If we get 100 of those problem-solving strategies all included, feels like we will see another hurdle cleared in terms of usefulness.
- deleted 2mo ago[deleted]
- andriy_koval 2mo ago> To be clear, we’ll want to see how this performs against the hold-out set. they could take open weight model, and check what will be impact from that harness on hold-out
- Zababa 2mo agoI think I'd typify it as "ARC-AGI doesn't matter" more than "harness matters". Or maybe "harness matters for some very specific tasks".
- YeGoblynQueenne 2mo agoARC-AGI is a bit of a joke at this point. The first version was supposed to be hard for neural nets because it had few examples and a super secret test set, with a different distribution than the public set no less. It took a while, but the original ARC ultimately fell to exactly the approach it was supposed to be protected from, i.e. big data memorisation, thanks to data augmentation techniques that increased the available training examples to the point that the private test set was eventually overcome. ARC-AGI 2 went the same way because it was basically the same kind of dataset except this time with some attempt to further defend it against LLMs with restrictions on the compute budget. And now ARC-AGI 3 is saturated within ... what is it, weeks? since its release. The fact that it's the public set that's beaten doesn't matter, when the score is 99%. Systems that can score ~90% on the public sets of the previous ARC's can comfortably reach 70-80% on the corresponding private test sets, as far as my eyballing of results suggests. It is time to accept that the whole idea of ARC is for the dustbin. It does not measure what it's supposed to measure -fluid intelligence, reasoning, whatever it is today. Its original premise, that a system could only beat ARC if it possessed human-like core knowledge systems (a-la Elizabeth Spelke's theory) has been comprehensively refuted: none of the systems that have ever performed well on any version of ARC has made any attempt to represent core knowledge systems in any way, shape or form. Ultimately, if your machine intelligence (let alone AGI) test relies on tricks like only giving a few examples or keeping a secret test set to defend itself against the dominant approach to machine intelligence... then it's not a useful machine intelligence test. Or it just doesn't measure machine intelligence but... something else. Who knows what.
- ubermon 2mo ago> Both scores come from a fixed fallback rule: Opus 4.8 and Sol xhigh run first; games scoring below 80 are rerun with Fable 5 and Sol max, respectively, and the higher per-game score is retained. hmm, this is like pass@n until you get the high watermark? How would this mean anything?
- pixl97 2mo agoDo you throw every problem at your most expensive programmer first? Or do you toss most things to your fleet of juniors and let them knock out the easy work and pass the harder problems to your senior developers. Because that's what real world problems look like and work like in business. Token costs are about the number one thing discussed by businesses these days.
- scotty79 2mo agoHow do humans get their results at these tasks? Don't they try until they succeed and claim their best run as their result?
- nkmnz 2mo agoWhere's the code?
- scotty79 2mo agoI'd bet gpt-5.6 could reimplement it in few days from this article.
- scotty79 2mo agoI pretty much predicted this. If such smart models capable of doing math research fail so hard on such simple games the interface is the problem, not the model. Right harness provides a good interface between the problem and the intelligence.
- netinstructions 2mo agoI can't tell based on this article if the authors intend to submit this to Arc-AGI for the private / held-out set of games for a verified score. The final section sorta seems like they won't bother because Arc-AGI-3 is "now saturated"
- nathan_compton 2mo agoGod, who the fuck are they even writing this slop for? Other machines? Neat. Maybe even deeply interesting. Absolutely garbage write up.
- markisus 2mo agoIt's funny that the machine intelligences that arose out of the bitter lesson cracked this benchmark by meticulously modeling each individual case with rule based approaches. When humans tried making AI through rule based approaches, maybe we only failed because we couldn't type out the rules fast enough. Or we got burned out after writing the thousandth heuristic to fix the never ending edge cases. And we also got side tracked by concerns like maintainability, modularity, and code re-use. But I wonder if we take modern or near-future LLMs who never tire and write code faster than any human, could we make a frontier level GOFAI agent?