6 ms·
> Ban offline training/pretraining. Models must train from scratch after submission Previously this was considered impossible so rule. My model shows this is po
by foota 15d ago
> Ban offline training/pretraining. Models must train from scratch after submission
Previously this was considered impossible so rule. My model shows this is possible
Guarantees no synthetic data can be used
It makes the comparison fair across differet models. Otherwise some models like LLMs can benchmaxx ARC by using ungodly amounts of offline training. (Since the benchmark has been around a long time, many ARC-like datasets have been created)
I'm not an ML researcher, so YMMV, but... how could a model learn to answer these ARC-AGI questions without training beforehand?
- nicman23 15d agohow would you as a human know the answer to the arc-agi questions?
- foota 15d agoI spent years learning logic and doing puzzles. I don't think a baby or even average kid could solve these.
- anon291 15d agoPredicate logic is trivially realized by linear transformations (I.e., matrices), and these matrices are easily discovered via gradient descent with appropriate reward functions. > I don't think a baby or even average kid could solve these The reward functions of a typical baby or kid is not 'get a huge dopamine boost when you solve a logic puzzle' (or whatever neurotransmitter, I don't know).
- evilmathkid 15d agotrain from scratch only during the 12 hours allowed on Kaggle Other competitions have implemented things like this before. Eg: OpenAI's Parameter Golf and Keller Jordan's Modded NanoGPT Speedrun
- foota 15d ago[dead]