7 ms·
A Mathematical Framework for Transformer Circuits (2021)
- myself248 4d agoNothing about B-H curves or magnetization current, must be that other kind of transformer, in that other kind of circuit.
- greenbit 4d agoSomeone should step up and rectify that, there's a lot of potential for confusion.
- ziofill 4d agoThis comment is hysteresical
- Bluestein 4d agoI find it polarizing.-
- thatspartan 4d agoIt can be quite eddyfying
- amelius 4d ago> A previous project, the Distill Circuits thread, has attempted to reverse engineer vision models, but so far there hasn’t been a comparable project for transformers or language models. How successful was that?
- Bluestein 4d agoThe entire thing went down: - https://distill.pub/2021/distill-hiatus/ https://distill.pub/2021/distill-hiatus/ ... and, here we are.-
- oofbey 4d agoI’ve tried reading this many times. But it’s sooooo long. Is it worth it?
- Bluestein 4d agoDepending on your field, degree of interest, general curiosity, its a must. Its up there with word2vec.-
- robrenaud 4d agoNeal Nanda's walkthrough of it is pretty good. https://youtu.be/KV5gbOmHbjU?is=GSiv0rSHBocNdU8l https://youtu.be/KV5gbOmHbjU?is=GSiv0rSHBocNdU8l Also long, but it's detailed and complicated, so there is no escaping that.
- _jayhack_ 4d agoShocking to me how little interest the general public has in mechinterp given the alien capabilities demonstrated by LLMs - this and the subsequent transformer-circuits.pub publications will be seen as classic, foundational work in a few years
- libraryofbabel 4d agoThis is one of those papers that ought to have several textbook chapters unpacking some of the insights... I'll just pick a couple things I love: 1) There's a kind of "rabbit–duck illusion" moment where they show how you can reframe all the linear algebra around attention in a totally different way, demoting the Q, K, and V matrices in favor of emphasizing a set of much larger matrices that are mathematically equivalent and very useful for interpretability purposes. (So, don't trust anyone who insists that vague analogies about "keys" and "queries" are the only good way to understand attention! They probably haven't read this paper.) 2) They show the importance of the "residual stream" inside LLMs: it's not just a series of bypasses of layers that's useful for training stability (like I first had it explained to me) but a kind of main communication bus that runs unbroken through the whole model from embedding to output. Attention heads and Feedforward are off to the side, adding embeddings into the stream. Somehow I find this a much more satisfying way to understand LLM architectures, and whenever I see an architecture diagram now I mentally redraw it with the residual stream at the center.