11 ms·
A walk through of the DeltaNet family of linear attention variants
- scarmig 2mo agoI like the math vs physics toggle.
- _davide_ 2mo agoLoved this incremental evolution, things gets way more understandable...usually xD
- rekshaw 2mo agoafter a cursory read, I can confidently say I could not, in fact, have come up with Kimi Delta Attention.
- frankus 2mo agoRelevant xkcd: https://www.explainxkcd.com/wiki/index.php/2501:_Average_Familiarity https://www.explainxkcd.com/wiki/index.php/2501:_Average_Fam...
- world2vec 2mo agoNot even close for me too.
- vovavili 2mo agoI thought I was the only one.
- trollbridge 2mo agoThank goodness. There are dozens of us.
- ma-r-s 2mo agodozens!!!
- arnavpraneet 2mo agopossibly scores!!!
- cloudcalvin118 2mo agoScores? More like "grosses"
- nope1000 2mo agoI don't even know most words they used in the paper haha
- dd8601fn 2mo agoYeah, pretty sure the “you” in “you could have” is a different “you” than “we”.
- queenkjuul 2mo agoThey lost me just describing the notation lol
- denysvitali 2mo agoSame!
- londons_explore 2mo agoThe notation looks complex, but underneath it's all just adding and multiplying. Nothing complex
- baq 2mo agoas is practically all of transformer maths if you squint hard enough...
- ReactiveJelly 2mo ago"I invented a new algorithm" "New algorithm, or fmadd?" "... fmadd."
- teach 2mo agoGrand Theft Auto VI looks complex, but underneath it's all just ones and zeros and NAND
- deleted 2mo ago[deleted]
- penguin_booze 2mo ago"you could have..." is among the top insulting phrases used by maths-adjacent people. Others in that league are "it should now be obvious...", "it's abundantly clear...", "it can be easily shown that...", "this is nothing but..." etc. The rest of us reading this are like, holy batman, what the fuck was that?!
- Razengan 2mo agoRight next to "Learn More" by software UI designers.
- ozgung 2mo agoAlso the proof is so trivial that it’s left to the reader.
- BurnerOptical 2mo agoThis one hurts the most, esp. in fields you're not familiar.
- hnfong 2mo agoWell, at least these days an actually trivial (to a domain expert) proof can be delegated to a frontier model...
- cubefox 2mo agoIn the future this might be replaced with "you can verify this fact by asking an LLM of your choice". Similar to how people in chat arguments already post screenshots of an LLM answer to show that their opinion is correct.
- moffkalast 2mo agoCounterexample left as an exercise for Fable
- egeozcan 2mo ago
- yongjik 2mo agoImagine reading the title again in the voice of the Asian Father Meme. "You could have come up with Kimi Delta Attention, but you didn't, did you."
- devy 2mo agoDoubleword AI is conducting a classic textbook marketing trick called newsjacking. Writing a detailed technical post behind the news of Kimi K3 and KDA algorithm with an audacious title like "You Could Have Invent Breakthrough It too" they are pre-filtering out the ones who couldn't comprehend with quick read (myself included) and attracting the ones who agreed with the blog post. At the end with a strong CTA to promoting their 10x cheapter open weight model AI inference and hiring too. Good job Doubleword, I see what you are doing there.
- Barbing 2mo ago“Kimi Delta Attention” because “Kimi K3 Delta Attention (oh that’s just our little internal name for it as a joke)” passes no sniff tests.
- mezark 2mo agolol - co-founder of Doubleword here. honoured you think we have sophisticated enough marketing to 'newsjack'. What actually happened is my cofounder wrote it over the weekend because he's a mega-nerd and put it live yesterday. I hadn't even read it until I saw this hacker news thread lol We're just a group of guys and gals who like inference!
- arkj 2mo agoThe blogs from doubleword are indeed full precision. A great team doing a fantastic job.
- glaslong 2mo ago* a completely different "you" who spent countless hours gaining expertise on a wholly diverged life path
- myshapeprotocol 2mo ago[flagged]
- mnky9800n 2mo agowhy are you using braket notation?
- brcmthrowaway 2mo agoI could never get this about modern machine/deep learning or even the Transformers. Yes, it's not exactly rocket science, but when I see the data flow diagrams, it's not clear what is calculated in real time or multiple steps. Is it really one big computation f(g(h(x)))?
- leonvoss 2mo agoIt's all vibes.
- deleted 2mo ago[deleted]
- malwrar 2mo agoYes. Each token prediction is one big function call. Then you just recursively generate more tokens until run out of context or the model predicts a next token indicating end of sequence. Technically the model outputs a matrix where the last row is a probability distribution, but I’m counting sampling from it as part of the chain. Hundreds of billions of dollars has gone into just making the function fatter and gradually changing pieces here and there.
- brcmthrowaway 2mo agoI remember the concept of layers, as essentially defining the matrix math dimensions. And for a given model/framework, they were static. That always bugged me (not very dynamic).. is this still the case?
- malwrar 2mo agoThe sizes for the matrices are decided up front (“hyperparameters”, more or less guesstimated) and the values that are learned (“parameters”, for completeness) and remain static after training. The layering just refers to the core “transformer block” being chained inside the model that contains most of the weights. Those themselves are pretty simple, usually just an MLP followed by an attention function, with variations on formulation & “multi-heading” in most of the new models. Behind the terms and math and inner architectural choices, it’s pretty much still the same GPT pattern of ‘“lie” = decode(sample(lm_head(T_1(T_2(…T_n(encode(“the cake is a”))))))’, with T_i(…) being the ith layer. You know, I haven’t kept up with MoE and etc where there’s a bit of selection going on, so I should probably be more humble. I think new work has only added different “paths down the same hill” though (no recurrence, just select different matrices), but could be wrong there. I don’t think I’m wrong on my general intuition, just want to be epistemically honest!
- neutrinobro 2mo agoYou know its a doozy when the author writes a disclaimer at the top saying that bra-ket notation was chosen in order to make the algorithm and data structures clearer.
- CodesInChaos 2mo agoOne of the more annoying parts of my physics study was getting used to the new matrix multiplication notation they came up with every semester.
- kurthr 2mo agobra-ket is the (most?) general form of tensor manipulation. Raising and lowering operators for summation notation are the beginner tools for covariant derivatives of the metric tensor. Christoffel symbols are where it's at, if you need to write out the Ricci tensor. The more constrained the space the more concise the notation can be. Note that MechE tensor notation has an even more compact (eigen) form for principal stresses.
- LogicFailsMe 2mo agoAll of this is true, but I don't believe and I want to be wrong about this that there is something in this notation that starts at an ELI 5 level and gently guides you to physicist level expertise. I all but majored in math (deriving back prop was trivial once it was clear it was the chain rule as one example) but I have never been able to keep bra ket notation straight in my head for the more exotic operations. Einsteinian notation on the other hand is a few minutes of furled brows and then all is clear. It is what has separated me from being able to code just about anything on a GPU and being known for some of that work and coming up with a better way to run ab initio quantum chemistry on them. It truly has been my Waterloo for many years. So make me wrong.
- kurthr 2mo agoYeah, bra-ket is arbitrary tensors (inner and outer multiplication) rather than the nice 4D of space-time (with derivatives). I will say that seeing transformers written this way gives me a bit more intuition for what is going on (being able to identify correct equations), but there's enough complexity in actual transformer implementations, that it still feels like I'm fooling myself. Conceivably, I think you could use Feynman diagrams to talk about phonon dispersion in (eg asymetric crystaline) solids, but even though they're a "simplification", they're overkill for the problem.
- Kushagra125 2mo agoThe toggle is really useful. Liked it!!
- spwa4 2mo agoNo, you couldn't have. There are plenty of ML innovations that when push comes to shove only depend on having access to more compute, but this is one of the worst examples I've ever seen. I always thought that the jump from LSTM/GRU -> Attention wasn't a particularly big one. Instead of partial unroll, do a full unroll. Why not (because it's too expensive, that's why not). Every component was known, and everybody anywhere near ML knew perfectly well why NOT to try that: because you just don't have the compute to fully unroll an LSTM. From that point attention is optimized (they key-query mechanic). The big innovation is not so much the mechanism itself but realizing the parallelize-ability of it. It's sort of like if one would today make the "improvement" to attention to replace they key-query-value mechanic by just dropping it while making the entire context the latent space. That will outperform attention, nearly guaranteed. It'll also make even Google's cluster networks meltdown. Attention is one of those innovations that came mostly from realizing you had better hardware than everybody else and asking yourself how to use it. It's still quite the accomplishment, they had to get it working. But nobody else was really capable of making this leap.
- leonvoss 2mo agoI agree 100%. This field is not amenable to progress from people with a pen sitting in a corner proving theorems. The math is mostly uncertain vibes and to test it you need millions of dollars of compute. Smart loners just can't.
- p1esk 2mo agoreplace they key-query-value mechanic by just dropping it while making the entire context the latent space. What do you mean by this? Like concatenating all token embeddings into one large vector?
- spwa4 2mo agoYes. Just have the entire context visible to everything, all the time. It's just one way to increase the expressiveness of the whole network. In general one could say that coming up with ways to greatly increase the expressiveness of neural nets is not hard. Or it's hard because it blows up compute. Meanwhile the human mind demonstrates that you can drop compute by 5-6 orders of magnitude without losing expressiveness to save power.
- asdfman123 2mo agoRelevant XKCD https://xkcd.com/2501/ https://xkcd.com/2501/
- TrackerFF 2mo agoMachine learning could need, and probably has needed, some unified math notation for the past 15 years IMO. With that said, it was worse back in the day - when ML papers were the products of researchers from all over, you'd see some wild notation. Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the very start...that is not always the case. Rarely, even. EDIT: Didn't even notice the notation switch, much appreciated.
- olalonde 2mo agoI never understood people who preferred traditional math notation (e.g. single letter symbols, weird characters like ∣q⟩ instead of writing down an explicit type, etc.). I guess the main advantage is terseness? To me, the mathematical expressions would be so much easier to understand if they were just written in pseudo code or an actual programming language like Python.
- htrp 2mo agomath notation doesn't bias towards English language understanding like pseudocode
- piterrro 2mo agoAt first I felt bad about not having come up with this solution. But then I realized I have problems with writing binary search by myself in JS and immediately felt better. Now way I could have come up with Kimi Delta Attention.
- bee_rider 2mo agoLots of linear algebra codes are actually “easy to write” in a way. It isn’t like conventional CS where you are always going a bunch of recursive nonsense going on. There should be mathematical relationships between all of the variables, there are well implemented libraries for the common mathematical concepts, and it is rare to need to go more than a couple loops deep (anything more complex than that should get shunted off into a library anyway).
- croemer 2mo agoLLM written for sure: > The identity [...] is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state S_t.
- geraneum 2mo agoThis is what you get when you prompt claude to avoid –
- robertclaus 2mo agoYa, probably started with asking for a buzzy title.
- anshumankmr 2mo agohttps://www.youtube.com/watch?v=p0CEOkoSsgA https://www.youtube.com/watch?v=p0CEOkoSsgA
- _Microft 2mo agoSide note, before you ask: yes, bra-ket notation is called like that because of the brackets. https://en.wikipedia.org/wiki/Bra-ket_notation https://en.wikipedia.org/wiki/Bra-ket_notation
- andai 2mo ago>You Could Have Come Up With Kimi Delta Attention What? Little old me! Well, then, let's have a look... > (First paragraph) > A note on notation: this article defaults to bra-ket notation because (in my quantum-inspired opinion) it makes the shapes in this derivation very clear. The Math notation switch above rewrites every equation using conventional bold vectors and explicit transposes instead. In bra-ket mode, ∣ q ⟩ ∣q⟩ is a column vector, ⟨ k ∣ ⟨k∣ is a row vector, ⟨ k ∣ q ⟩ ⟨k∣q⟩ is a number, and ∣ v ⟩ ⟨ k ∣ ∣v⟩⟨k∣ is a matrix. Vectors face right by default, while keys face left when written into the linear-attention state. We work with one causal attention head and real-valued vectors, assume DeltaNet’s keys are normalized, and let the state map from key space to value space. Hmm... Guess not!
- 5555watch 2mo agoI love that they let you switch to a more common q'k notation!
- bee_rider 2mo agoWhere do linear algebra folks go to get started with ML stuff? It seems pretty easy but the hardware is expensive.
- nifets 2mo agowhat is a linear algebra folk?
- stuxnet79 2mo ago> It seems pretty easy but the hardware is expensive. Huh? If your aim is to truly 'get started' with ML then hardware is absolutely not a bottleneck (either local or cloud). Remember that ML is much more than LLMs. Even modern day LLMs can be quantized to a point where they can run on local hardware although their capabilities won't be as impressive. I would recommend looking into some of Andrej Karpathy's videos if you want a grasp of the basics.
- sva_ 2mo agoI think Karpathys nn zero to hero is a good starting point. And you can experiment on small networks using pretty normal hardware.
- thatjoeoverthr 2mo agoI’m having a great time with an NVIDIA 3090. 24 GB RAM will run a lot of neat models. But at zero you can for sure just do CPU until you build a project ambitious enough.
- codeduck 2mo agoHmm. Hmmm. Hmm. HMMM. Hmm. Yep! I know some of these words.
- HonshinM 2mo agoA visualized tutorial: https://snowchord.com/blog/linear-attention-visualized/ https://snowchord.com/blog/linear-attention-visualized/
- enraged_camel 2mo agoIf I could, I'd be working for one of the labs and commanding a seven-figure salary. :)
- juancn 2mo agoI really liked the ket notation. I was aprehensive at first, but it makes operations much more clear. I would have liked some refresher on some variables though (like d_k in quadratic attention).
- lain98 2mo agoIts greek to me.
- xp84 2mo agoWhen I see these types of articles and headlines, it just makes me supremely grateful for all the many people far smarter[1] than me. And humbles me, too, since I actually passed for a "very smart person" in places like high school and undergrad. In fact, I'm 'smart' for an average person, but there are definitely millions of people who make me look like a rube in comparison. [1] I specifically mean those who are able to hold very big complex ideas and systems in their head, and reason about them, which seems to be an important talent for mathematicians.
- abixb 2mo agoYes. I continue to believe that humans will still be the source of the vast majority of novel ideas, even as they increasingly use AI-related tools to accelerate their works. One of the though experiments I ran with one of my friends during a recent conversation over drinks was this: raising a bunch of "control group" kids away from the screens and the algorithmic ocean of "normie-tier content," and in a very learner-friendly setting with hyper-strict control on the quality of media and source material they get access to, just like we've been doing it with frontier models. Think of it like a monastery but for kids, while teaching them all the latest advances in our understanding of reality through mathematics, engineering, computer science, deep learning, and whatnot. What I'm getting at it is that we might still need super smart people to push the boundaries of knowledge while using super-advanced AI tools, and anyone who says AI will "completely replace" humans are just misguided. We will always need super smart people with largely unadulterated thinking.
- dmd 2mo agoYou should read 'Anathem' by Neal Stephenson, which goes into great deal about this kind of establishment.
- rhymeswithjazz 2mo agoI was reading their comment and was just about to suggest the same thing.
- 2mo ago
- nurettin 2mo agoIt is heartwarming to see how sarcasm turns into a celebration of mediocrity.
- sodapopcan 2mo agoOhhhhh Diag(αt), right. I was almost there but had left the placeholder "Diag(foo)" and never noticed. I now see is why I didn't come up with it first. So close!
- luciana1u 2mo ago[flagged]
- joe_the_user 2mo agoOverall, all the different linear attentions out there are approximations the original (quadratic) attention and this is important for the whole "AI" enterprise[2]. Original attention involves (very crudely) an approach of scanning how every token (roughly a word) relates every other token and training a classic neural network on related tokens - to get either language translation or next word prediction (and next word prediction is what "seems intelligent" in LLMs). [1] The problem is that since original attention is "everything to everything else" it scales quadratically (O(n^2)) with the size of the train set (or train set window) and so basically even the largest data center can use that once a truly vast training set is accumulated. Which is to say that "dirty little secret" of LLMs following the "Attention Is All You Need" paper don't actually scale. That model (in my crude, amateur understanding) is elegant for allowing every word's connection to every other word to be weighed and still brute-force for not starting with or achieving "understanding" of the words [3 give only some background but also why "full" attention is powerful]. Linear attention is a way around the quadratic quality of original attention so everyone is naturally using clever approaches to make it work. Simplifying terribly - you're trying to determine the value of word before you see in context. But my intuition is that since (Everything X Everything) is inherently a quadratic relationship, none of these can capture their expanded data set in the way original LLMs did - not they are worse but all the models seem likely to hit diminishing returns in terms of blindly capturing meaning from all-the-world's text (and data). Background and notes: [1] https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture) https://en.wikipedia.org/wiki/Transformer_(deep_learning_arc... [2] Linear Transformers Are Secretly Fast Weight Programmers: https://proceedings.mlr.press/v139/schlag21a/schlag21a.pdf https://proceedings.mlr.press/v139/schlag21a/schlag21a.pdf [3] Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machines: https://arxiv.org/pdf/2106.01506 https://arxiv.org/pdf/2106.01506
- krackers 2mo agoI had hoped the OP article would have gone into more depth on the intuition as to why you'd expect deltanets to work at all. It seems like we're going back to LSTMs and RNNs where you compress the history back into a fixed-size hidden state. Aside from the easy parallelization in training, I thought that Attention worked much better because it got over this fundamental bottleneck and just let every token interact with every past token (of course you pay for it in compute, but ultimately that's what got us a GPT). I guess that's probably why you still need some MLA layers in there.
- benjiro29 2mo ago> You Could Have Come Up with ... Creating or combining to have something new, that does not already exist is actually freaking hard! The moment its presented and people go "o, that is not that difficult", "i was able to also do that", or some nonsense like that. Everything looks simply the moment somebody did the hard work. We have all been there was developers. Thinking we invented something new, and ... then you discover somebody already made it in the 70's and its everywhere. But because it never cross your path, you never realized it existed.
- MrFiskarBengt 2mo agoYou could've also came up with Newton's laws. After all, they look trivial in retrospect. But, there's an important lesson in a story about balancing an egg here that can teach us something. Filippo Brunelleschi said he could build the large dome for the church that had stood unfinished for a century. Skeptical, other's demanded he'd explain how. He refused. Instead he challenged everyone to balance an egg on its tip. Nobody could do it. He then demonstrated by lightly tapping the egg on the table, flattening the tip, making it stand. "Anyone could've done that! You never said we could break the egg!". And that's the point. Anyone could've done it. But nobody did. Nobody thought 'outside the box'. And likewise, his solution to building the dome is as simple, and as ingenious. It's called Egg of Columbus. (there's a similar story about Columbus that's more famous, but apparently fictitious). It teaches us that hindsight is 20/20.
- edflsafoiewq 2mo ago"You could have invented..." a just a genre of expository writing which builds a path from something you know to something you want to learn. The sequence of development is not necessarily even historically accurate, it can be completely invented as long as the learner finds it natural and it helps motivate them. The fact that invention is hard and actually you probably couldn't have invented a bunch of hard stuff is not really that, um, relevant.
- dr_kretyn 2mo agoThe bra-ket notation makes this all very simple/intuitive for me. With "vectors" I always get confused which is horizontal/vertical, and then I just follow blobs, and get distracted, and leave. With bra-kets the whole thing was very intuitive! I'm now going to covert other articles to the notation as I must have missed a lot of good stuff! (Side notes: I have physics PhD and mild dyslexia)
- alex-moon 2mo ago"Machine learning architecture is second nature to us AI researchers, so it's easy to forget that the average person probably only knows the formulas for gradient descent and ReLU." "And softmax of course."
- krackers 2mo agoThere was a longform post on twitter which went through the same derivation at a bit higher level https://x.com/waterloo_intern/article/2081762065392541951 https://x.com/waterloo_intern/article/2081762065392541951 and in particular this image which clarifies the key essential difference between liner attention and delta network by examining the case of two tokens with same key but different value https://pbs.twimg.com/media/HOPCc7BaEAAQDtO.jpg?format=jpg&name=orig https://pbs.twimg.com/media/HOPCc7BaEAAQDtO.jpg?format=jpg&n... I think for comparison it would also have been good to have how original quadratic attention handles it: since both keys are identical, the attention would be "evenly divided" between both values so the final output would be the average of both values, as opposed to the latest value
- alf42red 2mo agoI've only read 1-4 without prior knowledge of qkv attention and I like the explanation. The only thing I didn't get from the text was why we divide by the square root of d_k, but ChatGPT explained it was to counter variance scaling with the dimension of the matrix. I probably couldn't have come up with this myself, but I feel that it actually makes sense now and I like that deltanet attention "learns a learning rate" according to one interpretation of beta from the text, if I'm understanding it correctly. Great article!