7 ms·

That’s why you have a private dataset.

by 10xDev 2mo ago

Jensson 2mo ago
Doesn't matter, people built harnesses that solves arc agi 3, so all you need is to train your model to work like that harness by default. That makes a model specialized at solving arc agi 3 without making it smarter in general. It is very hard to make a benchmark you can't do that for, but it is very easy to make your own personal test that others can't do that for since now it isn't a benchmark they can target.
- deleted 2mo ago
  [deleted]
- NitpickLawyer 2mo ago
  > people built harnesses that solves arc agi 3, They didn't. Kaggle is still running for a few more months, best result atm is ~2% with 9h runtime on one rtx6kPRO. Also note that these new results are on the semi-private set, not the public 25 games ones. Any announcement where you see "solved ARC3" is likely only dealing with the 25 public games. And that's highly questionable, until you get to see the code. (which, to my knowledge the team that claimed 99% hasn't yet published).
  - haldujai 2mo ago
    For frontier models, not local. https://schema-harness.github.io/ https://schema-harness.github.io/
    - NitpickLawyer 2mo ago
      Yes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.
      - Stevvo 2mo ago
        If you stop and think about the problem it really is quite simple. Just need to build a graph of the game state and then run A* to get to the end.
        NitpickLawyer 2mo ago
        You really should play the 25 games before stating that it's "simple". The benchmark doesn't just track "completion", it also tracks the number of steps, and the score is based on the median steps took by human players. So in order to get 99% it would mean that the model solved every level of every game in less steps than the median humans. Which, having played the games and having setup harnesses for local models, I find hard to believe. Also the models have to figure out what "end" means. And each game involves some kind of "gotchas" thrown in the harder levels. Some games are only solved by about 2/10 people trying them. The 99% result most likely has some leakage somewhere, either in the preparation of the environments, or from session to session. Seriously, play some of the games. They're fun.
        LogicFailsMe 2mo ago
        You are not given the rules or the winning conditions. You are only given a potentially windowed and/or degenerate visualizer of the underlying game state along with the UI and told to just figure it out. And you as a human will, in a couple moves. An LLM? Not so much. But they do eventually solve them. And given enough moves, victory is inevitable, but you are penalized for taking more moves than a human, yet also slightly punished if you find a better solution by capping your reward to 115%.
      - haldujai 2mo ago
        No but the session traces are available. It passes the sniff test considering how AGI-3 is scored and how this wrapper works. For example on bp35 it took fable 290M and >12k simulated turns for 566 real turns and finish more efficiently than a human. Regardless of the true score I think the takeaway is the benchmark measures the wrapper rather than the model. https://huggingface.co/schema-harness https://huggingface.co/schema-harness
raincole 2mo ago
Which you have sent to Anthropic/OpenAI/Google's servers when you run the benchmarks for the previous models.

That’s why you have a private dataset.

by 10xDev 2mo ago

That’s why you have a private dataset.

Doesn't matter, people built harnesses that solves arc agi 3, so all you need is to train your model to work like that harness by default. That makes a model specialized at solving arc agi 3 without making it smarter in general. It is very hard to make a benchmark you can't do that for, but it is very easy to make your own personal test that others can't do that for since now it isn't a benchmark they can target.

deleted 2mo ago
[deleted]
NitpickLawyer 2mo ago
> people built harnesses that solves arc agi 3, They didn't. Kaggle is still running for a few more months, best result atm is ~2% with 9h runtime on one rtx6kPRO. Also note that these new results are on the semi-private set, not the public 25 games ones. Any announcement where you see "solved ARC3" is likely only dealing with the 25 public games. And that's highly questionable, until you get to see the code. (which, to my knowledge the team that claimed 99% hasn't yet published).
- haldujai 2mo ago
  For frontier models, not local. https://schema-harness.github.io/ https://schema-harness.github.io/
  - NitpickLawyer 2mo ago
    Yes, saw that. They haven't yet released any code. Until they do, treat it with a huuuge grain of salt. In fact treat any 99% result in ML with a huge grain of salt.
    - Stevvo 2mo ago
      If you stop and think about the problem it really is quite simple. Just need to build a graph of the game state and then run A* to get to the end.
      - NitpickLawyer 2mo ago
        You really should play the 25 games before stating that it's "simple". The benchmark doesn't just track "completion", it also tracks the number of steps, and the score is based on the median steps took by human players. So in order to get 99% it would mean that the model solved every level of every game in less steps than the median humans. Which, having played the games and having setup harnesses for local models, I find hard to believe. Also the models have to figure out what "end" means. And each game involves some kind of "gotchas" thrown in the harder levels. Some games are only solved by about 2/10 people trying them. The 99% result most likely has some leakage somewhere, either in the preparation of the environments, or from session to session. Seriously, play some of the games. They're fun.
      - LogicFailsMe 2mo ago
        You are not given the rules or the winning conditions. You are only given a potentially windowed and/or degenerate visualizer of the underlying game state along with the UI and told to just figure it out. And you as a human will, in a couple moves. An LLM? Not so much. But they do eventually solve them. And given enough moves, victory is inevitable, but you are penalized for taking more moves than a human, yet also slightly punished if you find a better solution by capping your reward to 115%.
    - haldujai 2mo ago
      No but the session traces are available. It passes the sniff test considering how AGI-3 is scored and how this wrapper works. For example on bp35 it took fable 290M and >12k simulated turns for 566 real turns and finish more efficiently than a human. Regardless of the true score I think the takeaway is the benchmark measures the wrapper rather than the model. https://huggingface.co/schema-harness https://huggingface.co/schema-harness

raincole 2mo ago

Which you have sent to Anthropic/OpenAI/Google's servers when you run the benchmarks for the previous models.