6 ms·

They're only allowed 2-3 guesses per problem. So even though yes it generates many candidates, it can't validate them - it doesn't have tool use or a verifier,

by tikkun 2y ago

They're only allowed 2-3 guesses per problem. So even though yes it generates many candidates, it can't validate them - it doesn't have tool use or a verifier, it submits the best 2-3 guesses. https://www.lesswrong.com/posts/Rdwui3wHxCeKb7feK/getting-50-sota-on-arc-agi-with-gpt-4o?commentId=r88fJFHsY9hAGPBAa https://www.lesswrong.com/posts/Rdwui3wHxCeKb7feK/getting-50...

nmca 2y ago
It is allowed exactly two guesses, per the ARC rules.
- trescenzi 2y ago
  How many guesses is the human comparison based on? I’d hope two as well but haven’t seen this anywhere so now I’m curious.
  - nmca 2y ago
    The real turker studies, resulting in the ~70% number, are scored correctly I believe. Higher numbers are just speculated human performance as far as I’m aware.
deleted 2y ago
[deleted]
TrapLord_Rhodo 2y ago
Chain of thought can entirely self validate. The OP is saying the LLM is acting like a photon, evaluate all possible solutions and choosing the most "Right" path. not quoting the OP here but my initial thought is that is does seem quite wasteful. the LLM only gets two guesses at the "end solutions". The whole chain of thought is breaking out the context, and levels of abstraction. How many "Guesses" is it self generating and internally validating, well that's all just based on compute power and time. My counter point to OP here would be is that is exactly how our brain works. In every given scenario, we are also evaluating all possible solutions. Our entire stack is constantly listening and eithier staying silent, or contributing to an action potential (eithier excitatory, or inhibitory). but our brain is always "Evaluating all potential possibilities" at any given moment. We have a society of mind always contributing their opinion, but the ones who don't have as much support essentially get "Shouted down".
- thomasahle 2y ago
  > How many "Guesses" is it self generating and internally validating That's completely fair game. That's just search.

by tikkun 2y ago

nmca 2y ago

It is allowed exactly two guesses, per the ARC rules.

trescenzi 2y ago
How many guesses is the human comparison based on? I’d hope two as well but haven’t seen this anywhere so now I’m curious.
- nmca 2y ago
  The real turker studies, resulting in the ~70% number, are scored correctly I believe. Higher numbers are just speculated human performance as far as I’m aware.

deleted 2y ago

[deleted]

TrapLord_Rhodo 2y ago

Chain of thought can entirely self validate. The OP is saying the LLM is acting like a photon, evaluate all possible solutions and choosing the most "Right" path. not quoting the OP here but my initial thought is that is does seem quite wasteful. the LLM only gets two guesses at the "end solutions". The whole chain of thought is breaking out the context, and levels of abstraction. How many "Guesses" is it self generating and internally validating, well that's all just based on compute power and time. My counter point to OP here would be is that is exactly how our brain works. In every given scenario, we are also evaluating all possible solutions. Our entire stack is constantly listening and eithier staying silent, or contributing to an action potential (eithier excitatory, or inhibitory). but our brain is always "Evaluating all potential possibilities" at any given moment. We have a society of mind always contributing their opinion, but the ones who don't have as much support essentially get "Shouted down".

thomasahle 2y ago
> How many "Guesses" is it self generating and internally validating That's completely fair game. That's just search.