shawntan

10 ms·

1.
by shawntan 6d ago
In the limited cases (below Turing completeness) there are properties of what can be done with O(log N) depth vs O(N) CoT (regular languages), if you look at the 2nd Will Merrill paper I referenced.
2.
by shawntan 7d ago
For the research focused, there are some references in my blogpost here on what kinds of computational problems minimally require how much CoT to solve: https://blog.wtf.sg/posts/2023-02-03-the-new-xor-problem/ No
3.
by shawntan 11mo ago
The question I keep coming back to is whether ARC-AGI is intended to evaluate generalisation to the task at hand. This would then mean that the test data has a meaningful distribution shift from the training data, and only a model that can
4.
by shawntan 11mo ago
This would not help if no proper constraints are established on what data can and cannot be trained on. And maybe just figuring out what the goal of the benchmark is. If it is to test generalisation capability, then what data the model bein
5.
by shawntan 11mo ago
You can have benchmarks with specifically constructed train-test splits for task-specific models. Train only on the train, then your results on test should be what is reported. You can still game those benchmarks (tune your hyperparameters
6.
by shawntan 11mo ago
This is a point I wish more people would recognise.
7.
by shawntan 11mo ago
I should probably also add: It's long been known that Universal / Recursive Transformers are able to solve _simple_ synthetic tasks that vanilla transformers cannot. Just check out the original UT paper, or some of it's follo
8.
by shawntan 11mo ago
> Do you mean that HRM and TRM are specifically trained on a small dataset of ARC-AGI samples, while LLMs are not? Or which difference exactly do hint at? Yes, precisely this. The question is really what is ARC-AGI evaluating for? 1. If
9.
by shawntan 11mo ago
Right. There should really be a vanilla Transformer baseline. With recurrence: The idea has been around: https://arxiv.org/abs/1807.03819 There are reasons why it hasn't really been picked up at scale, and the met
10.
by shawntan 11mo ago
That analysis provided a very non-abrasive wording of their evaluation of HRM and its contributions. The comparison with a recursive / universal transformer on the same settings is telling. "These results suggest that the performa
11.
by shawntan 11mo ago
I think everyone should read the post from ARC-AGI organisers about HRM carefully: https://arcprize.org/blog/hrm-analysis With the same data augmentation / 'test time training' setting, the vanilla Trans
12.
by shawntan 1y ago
Not sure if you mean in general, but I'll answer both branches of the question. In general: Depending on the method of compression, you can have lossy or non-lossy compression. Using 7zip on a bunch of text files can lossless-ly compre
13.
by shawntan 1y ago
The compression is lossy.
14.
by shawntan 1y ago
Sup!
15.
by shawntan 1y ago
2nd employee at Semantics3 here. Considering all the AI available today I think things like product disambiguation becomes wayyy easier. We were trying many tricks and heuristics to identify the same products across sites.
16.
by shawntan 1y ago
Systems might want to anticipate changes in LLM architectures (even small changes can make a big difference kernel wise), so it's good to not "bake" too much in ahead of time. That said, at some point it just depends where th
17.
by shawntan 1y ago
I'm curious how the speed is achieved is this is the technique used. Generally I expected this "masked language model" technique to be far slower since the full vocab projection needs to be computed every iteration. I always
18.
by shawntan 2y ago
The formulations in attention as rnn have similar issues as rwkv. Fundamentally it's a question of what we call an RNN. Personally I think it's important not to call some of these recent architectures RNNs because they have theore
19.
by shawntan 2y ago
Although marketed as such, RWKV isn't really an RNN. In the recent RWKV7 incarnation, you could argue it's a type of Linear RNN, but past versions had an issue of taking its previous state from a lower layer, allowing for parallel
20.
by shawntan 2y ago
> The actual result of the paper is that any poly-time computable function can be computed with poly-many tokens. You're right. Re: NAND of two inputs. Isn't this doable even by a single layer (no hidden layers) neural network?
21.
by shawntan 2y ago
If a "problem we care about" is not stated as a formal language, does it mean it does not exist in the hierarchy of formal languages? Or is it just as yet unclassified?
22.
by shawntan 2y ago
Using CoT implicitly increases the depth of the circuit. But yes, poorly worded.
23.
by shawntan 2y ago
You can't really be blamed though, the language in the paper does seem to state what you originally said. Might be a matter of taste but I don't think it's quite accurate. The prior work they referenced actually did account f
24.
by shawntan 2y ago
Generally, literature on the computational power of the SAME neural architecture can differ on their conclusions based on their premises. Assuming finite precision will give a more restrictive result, and assuming arbitrary precision can gi
25.
by shawntan 2y ago
Theoretical results exist that try to quantify the number of CoT tokens needed to reach different levels of computational expressibility: https://arxiv.org/pdf/2310.07923 TL;DR: Getting to Turing completeness can requi
26.
by shawntan 2y ago
There's a general trap people working on deep learning tend to fall into, thinking "Why don't we learn the activation function as well?" The answer to that really should be that a combination of linear and non-linear act
27.
by shawntan 2y ago
Yes this is a good empirical study on the types of tasks that's been shown to be impossible for transformers to generalise on. With both empirical and theoretical support I find it's pretty clear this is an obvious limitation.
28.
by shawntan 2y ago
I agree it's a good question to be asking. There are good answers to be found if you look. It feels like no proper looking was attempted.
29.
by shawntan 2y ago
> Neural nets can approximate any function. Common misunderstanding of the universal approximation theorem. Consider this: can an mlp approximate a sine wave? > A large enough llm with memory is turning complete. With (a lot of) chain
30.
by shawntan 2y ago
Not sure what you got out of the paper, but for me it was more spurring ideas about how to fix this in future architectures. Don't think anyone worth their salt would look at this and think : oh well that's that then.

More ›