colah3

7 ms·

1.
by colah3 11mo ago
(Disclaimer: I work on interpretability at Anthropic.) I wanted to flag that this is an accessible blog post and that there's a link to the paper ( https://transformer-circuits.pub/2025/introspection/index.ht.
2.
by colah3 1y ago
Hi! I'm the research lead for Anthropic's interpretability team, and was the decision maker for us publishing our papers web first and not doing traditional publications. A few thoughts: (1) As others have commented, I think peer
3.
by colah3 1y ago
See https://transformer-circuits.pub/2022/toy_model/index.html#m... If you're new to this, I'd mostly just look at all the empirical examples. The slightly harder thing is to consider the fact that neura
4.
by colah3 1y ago
It's a bit different than what's discussed here, but color-contrast detectors in neural networks can be thought of as forming a Klein bottle: https://distill.pub/2020/circuits/equivariance/#hue-rotat
5.
by colah3 1y ago
I guess I'll plug my hobby horse: The whole discourse of "stochastic parrots" and "do models understand" and so on is deeply unhealthy because it should be scientific questions about mechanism, and people don't
6.
by colah3 1y ago
> True! I suppose I was thinking about a 'strong' form of linear representations, which is something like: features are represented by linear combinations of neurons that display the same repulsion-geometries as observed in Toy
7.
by colah3 1y ago
If you like symmetry, you might enjoy how symmetry falls out of circuit analysis of conv nets here: https://distill.pub/2020/circuits/equivariance/
8.
by colah3 1y ago
> Circuits I find less compelling, since the analysis there feels very tied to the transformer architecture in specific, but what do I know. I don't think circuits is specific to transformers? Our work in the Transformer Circuits th
9.
by colah3 1y ago
Since this post is based on my 2014 blog post ( https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/ ), I thought I might comment. I tried really hard to use topology as a way to understand neural networks,
10.
by colah3 1y ago
A few comments on this thread: Gwern is correct in his prior quote of how long these articles took. I think 50-200 hours is a pretty good range. I expect AI assistants could help quite a bit with implementing the interactive diagrams, which
11.
by colah3 1y ago
Yep, that’s right! If you want to be precise, there are “autoregressive transformers” and “bidirectional transformers”. Bidirectional is a lot more common in vision. In language models, you do see bidirectional models like Bert, but autoreg
12.
by colah3 1y ago
Thanks for the great questions! I've been responding to this thread for the last few hours and I'm about to need to run, so I hope you'll forgive me redirecting you to some of the other answers I've given. On whether the
13.
by colah3 1y ago
Thanks for the feedback! I'm one of the authors. I just wanted to make sure you noticed that this is linking to an accessible blog post that's trying to communicate a research result to a non-technical audience? The actual researc
14.
by colah3 1y ago
> The obvious way to deal with this would be to send forward some of the internal activations as well as the generated words in the autoregressive chain. Hi! I lead interpretability research at Anthropic. That's a great intuition, a
15.
by colah3 1y ago
Hi! I'm one of the authors. There certainly are many interesting parallels here. I often think about this from the perspective of systems biology, in Uri Alon's tradition. There are a range of graphs in biology with excitation and
16.
by colah3 1y ago
Features correspond to vectors in activation space. So you can just do vector arithmetic! If you aren't familiar with thinking about features, you might find it helpful to look at our previous work on features in superposition: - http
17.
by colah3 1y ago
I think the question is: by what mechanism does it adjust up the probability of the token "an"? Of course, the reason it has learned to do this is that it saw this in training data. But it needs to learn circuits which actually
18.
by colah3 1y ago
Just to be clear, the probability for "An" is high, just based on the prefix. You don't need to do beam search.
19.
by colah3 1y ago
The planning is certainly performed by circuits which we learned during training. I'd expect that, just like in the multi-step planning example, there are lots of places where the attribution graph we're observing is stitching tog
20.
by colah3 1y ago
I used the astronomer example earlier as the most simple, minimal version of something you might think of as a kind of microscopic form of "planning", but I think that at this point in the conversation, it's probably helpful
21.
by colah3 1y ago
"An astronomer" is two tokens, which is the relevant concern when people worry about this.
22.
by colah3 1y ago
Yes, there are two kinds of evidence. Firstly, there is behavioral evidence. This is, to me, the less compelling kind. But it's important to understand. You are of course correct that, once Cluade has said "An", it will be in
23.
by colah3 1y ago
Hi! I lead interpretability research at Anthropic. I also used to do a lot of basic ML pedagogy ( https://colah.github.io/ ). I think this post and its children have some important questions about modern deep learning and how
24.
by colah3 2y ago
I'm the research lead of Anthropic's interpretability team. I've seen some comments like this one, which I worry downplay the importance of @leogao et al's paper due to the similarity of ours. I think these comments are
25.
by colah3 4y ago
I'm glad you've found it easy to follow! My best guess at the middle regime is that there are _empirical correlations between features_ due to the limited data. That is, even though the features are independent, there's some
26.
by colah3 4y ago
Gosh, that's very flattering! Very touched by your interest.
27.
by colah3 4y ago
I'm glad you've enjoyed it! If you like the idea of a periodic table of features, you might like the Early Vision article from the original Distill circuits thread: https://distill.pub/2020/circuits/early
28.
by colah3 5y ago
Thanks for the kind remark! > I don't think you should feel bad for being slow, or for doing "few" things at all. Unfortunately, I think it's tricky to do this in a journal format. If you accept submissions, you'
29.
by colah3 5y ago
It's certainly true that there are strong biological analogies. The analogy between first layer conv features and neuroscience is pretty widely accepted -- a lot of theoretical neuroscience models produce the same features.(It's l
30.
by colah3 5y ago
That's an interesting hypothesis which hadn't been on my radar. (I'm one of the authors.)

More ›