evilmathkid

6 ms·

1.
by evilmathkid 15d ago
well not if its an open textbook exam in non metaphor terms: In many ML situations you can carry the train set with you test time. Eg: KNN, SVM, replay buffers, etc this is one such case -- the separate overfitting concern is fair, look at
2.
by evilmathkid 15d ago
Yeah agree with pg completely. Multiple people I know did this
3.
by evilmathkid 15d ago
Yeah but I posted this work back in Dec-Feb lol
4.
by evilmathkid 15d ago
no, most models on kaggle are finetuning during test time, (including LLM based approaches) Pure frontier LLMs dont, but thats because nobody knows how to make it work cleanly and at scale. Once someone makes it work, it will be deployed
5.
by evilmathkid 15d ago
Not true. This is allowed in a metalearning context. Its called transductive learning and has existed since the 90s: https://en.wikipedia.org/wiki/Transduction_(machine_learning... I address this in more detail in the
6.
by evilmathkid 15d ago
I didn't phrase that well and cant edit, so clarifying: What I did was "You are born during the exam, given access to a training set (which is curated and allowed) and the questions then learn everything from scratch during the e
7.
by evilmathkid 15d ago
what you describe would be cheating. My approach is the opposite. What I did was "You are born during the exam, given access to a training set and the questions then learn from scratch during the exam " I put more details in the
8.
by evilmathkid 15d ago
2 different things are being called leakage here 1) weight update during eval: this is a form of test time training and not really cheating. It is also closer to Sutton's views of intelligence: models should learn during deployment, in
9.
by evilmathkid 15d ago
haha thanks!
10.
by evilmathkid 15d ago
As of today yes I agree with you in the future, for general perf, I am optimistic that someone will figure out an alphazero like approach (ilya/silver/sutton/carmack seem to be working on something like this)
11.
by evilmathkid 15d ago
train from scratch only during the 12 hours allowed on Kaggle Other competitions have implemented things like this before. Eg: OpenAI's Parameter Golf and Keller Jordan's Modded NanoGPT Speedrun
12.
by evilmathkid 15d ago
Glad to know you like it! 1) Unfortunately I didn't. I was v new to ML when I did this and didnt have time or skill to try many things. Will try them when I get some time! 2) Possibly, but it would require significant changes and effor
13.
by evilmathkid 15d ago
> You mean for a specific usecase? Yepp > Also aren't frontier models trained 'from scratch'? The full sentence was supposed to be "training from scratch only on ARC data" the point was you dont need large
14.
by evilmathkid 15d ago
> In the university I first dropped out of, students that surpassed me studied by getting and sharing copies of previous exams and solving those question Yes what you describe would be cheating. My approach is the opposite. What I did
15.
by evilmathkid 15d ago
same! I'd often learn during the exam by solving an easier problem and then that would let me tackle a hard problem
16.
by evilmathkid 15d ago
Thanks! I think this is a great question. I have some thoughts on this but no hard evidence (neither does anyone else!) Your argument relies on the AGI system being the model arch + weights. I think that the weights are irrelevant. The trai
17.
by evilmathkid 15d ago
there's a very large variance in doctors' abilities in India. At the very top they are close to the best in the world, esp with an insanely high workload. but on an average, not great Also, a lot of gymgoers and physical trainers
18.
by evilmathkid 15d ago
9 color tokens + 4 special tokens (start, end, newline, inp_out_sep)
19.
by evilmathkid 15d ago
Nope, it never learns how it did on the questions. During test time, you have to submit all the answers at once and you get the total score (so you dont even know which puzzles were solved)
20.
by evilmathkid 15d ago
thanks! was incredibly scary when it happened
21.
by evilmathkid 15d ago
I'm currently 10th in the world on the private set on Kaggle. And iirc, at one point I was 4th Can't comment more since its an ongoing competition
22.
by evilmathkid 15d ago
What you gather is correct, assuming by "the test" you mean the ARC benchmark in general. It was controversial because people are used to LLMs which are frozen at train time, where the eval problems are usually not trained on for
23.
by evilmathkid 15d ago
idk the definition is fuzzy. thats why people use the "modern" qualifier to talk about decoder-only style and this is also not clean since you now have reasoning models which are separate
24.
by evilmathkid 15d ago
its not gonna do well on ARC-3 without some significant changes and effort The new arch in that video is kinda misleading. Didn't really compare against proper baselines
25.
by evilmathkid 15d ago
> The "67 cents" part though is misleading imho. You can't extrapolate from there and think that investing say $100 will get you a lot better results. I think thats unfair. Perf-compute is often logarithmic and will always
26.
by evilmathkid 15d ago
Yeah I've reached much higher perf but - it feels logarithmic (like most perf-compute graphs), and eventually plateaus. 44% @ 67 cents was a good stopping point for me - more compute would require a lot of effort and dealing with new p
27.
by evilmathkid 15d ago
Its not an LLM if there's no pretraining. AR transformers were around before LLMs and will be there after LLMs. When I made this, the point was to show that you dont need pretraining (which is what makes an LLM) to perform well on comp
28.
by evilmathkid 15d ago
Hi! Author here. Surprised to see this on HN now. Happy to answer any questions! Some context about this: - This is NOT an LLM. its a small ar transformer trained from scratch. One of the points was that extremely complex problems can be ta
29.
44% on ARC-AGI-1 in 67 cents(github.com)
1 points by evilmathkid 7mo ago|1 comments
30.
by evilmathkid 7mo ago
This is a small transformer trained from scratch in 1.5hrs on a 5090 that beats many LLMs. Code is open source. I want to solve sample efficiency and this work is an attempt to find the limits of transformers and today's methods while

More ›