9 ms·

Understand how transformers work by demystifying the math behind them

470 points by LaserPineapple 3y ago

leereeves 3y ago
> The complexity comes from the number of steps and the number of parameters. Yes, it seems like a transformer model simple enough for us to understand isn't able to do anything interesting, and a transformer complex enough to do something interesting is too complex for us to understand. I would love to study something in the middle, a model that is both simple enough to understand and complex enough to do something interesting.
- PeterisP 3y ago
  I would assume that the boundaries of those ranges are such that the middle in between those extremes is something that is already too complex for a human to properly understand while still too small to be able to do anything interesting.
- calebkaiser 3y ago
  You might be interested, if you aren't already familiar, in some of the work going on in the mechanistic interpretability field. Neel Nanda has a lot of approachable work on the topic: https://www.neelnanda.io/mechanistic-interpretability https://www.neelnanda.io/mechanistic-interpretability
  - leereeves 3y ago
    I was not familiar with it, and that does look fascinating, thank you. If anyone else is interested, this guide "Concrete Steps to Get Started in Transformer Mechanistic Interpretability" on his site looks like a great place to start: https://www.neelnanda.io/mechanistic-interpretability/getting-started https://www.neelnanda.io/mechanistic-interpretability/gettin...
quickthrower2 3y ago
Transformer tutorials might be the new monad tutorial. A hard concept to get, but one you need to struggle with (and practice some examples) to understand. So a bit like much of computer science :-).
- hdhfjkrkrme 3y ago
  The moment you understand the Transformer you become incapable of explaining it.
- amelius 3y ago
  Waiting for a blogpost titled "You could have invented transformers".
- csdvrx 3y ago
  > Transformer tutorials might be the new monad tutorial. A hard concept to get, A hard concept? But a monad is just a monoid in the category of endofunctors, so what's the problem?
nemo8551 3y ago
There I was all excited to show off some of my electrical chops on HN. Not today.
- rzzzt 3y ago
  Does mystified math lie beyond behind how the ratio of input and output voltages is equal to the ratio of the primary and secondary windings? Can it be derived from Maxwell's equations? Off to a search...
  - amelius 3y ago
    I bet that an LLM (which uses transformers) can explain those aspects of a transformer to you.
    - wardedVibe 3y ago
      Or just read Wikipedia, which I'm sure it will crib from poorly... https://en.wikipedia.org/wiki/Transformer https://en.wikipedia.org/wiki/Transformer
    - nemo8551 3y ago
      I seen an LLM, or maybe another variant of “AI” [0] a while back that could aid design of electronic circuits by having a pool of data sheets added for referencing. As you were querying specs for a board at component level it could give you a schematic, I think, with citations to the actual data sheets. I suppose the same scale up could be used for systems that needed a varying number of specific power supplies. [0] https://www.flux.ai/p https://www.flux.ai/p
      - amelius 3y ago
        Actually, in my experience chatgpt knows quite a lot about electronic components so I wouldn't be surprised if it was fed datasheets.
        3y ago
ActorNightly 3y ago
The whole "mystery" of transformer is that instead of a linear sequence of static weights times values in each layer, you now have 3 different matrices that are obtained from the same input through multiplication of learned weights, and then you just multiply the matrices together. I.e more parallelism which works out nice, but very restrictive since the attention formula is static. We arent going to see more progress until we have a way to generalize the compute graph as a learnable parameter. I dunno if this is even possible in the traditional sense of gradients due to chaotic effects (i.e small changes reflect big shifts in performance), it may have to be some form of genetic algorithm or pso that happens under the hood.
- ex3ndr 3y ago
  This is basically this - it can learn ignore some paths, and amplify something more important, then you can just cut this paths without sensible loss of quality. The problem is that you are not going to win anything from this - non-matrix multiplication would be slower or the same.
  - ActorNightly 3y ago
    The issue is that you are thinking of this in terms of information compression, which is what LLMs are. Im more concerned with an LLM having the ability to be trained to the point where a subset of the graph represents all the nand gates necessary for a cpu and ram, so when you ask it questions it can actually run code to compute them accurately instead of offering a statistical best guess, i.e decompression after lossy compression.
    - exe34 3y ago
      Just give it a computer? Even a virtual machine. It can output assembly code or high level code that gets compiled.
      - ActorNightly 3y ago
        The issue is not having access to the cpu, the issue is that the model being able to be trained in such a way that it has representative structures for applicable problem solving. Furthermore, the structures itself should Philosophically, you can start ad hoc-ing functionalities on top of LLMs and expect major progress. Sure, you can make them better, but you will never get to the state where AI is massively useful. For example, lets say you gather a whole bunch of experts in respective fields, and you give them a task to put together a detailed plan on how to build a flying car. You will have people doing design, doing simulations, researching material sourcing, creating CNC programs for manufacturing parts, sourcing tools and equipment, writing software, e.t.c. And when executing this plan, they would be open to feedback for anything missed, and can advise on how to proceed. The AI with above capability should be able to go out on the internet, gather respective data, run any soft of algorithms it needs to run, and perhaps after a month of number crunching on a cloud rented TPU rack produce step by step plan with costs on how to do all of that. And it would be better than those experts because it should be able to create a much higher fidelity simulations to account for things like vibration and predict if some connector if going to wobble loose .
brcmthrowaway 3y ago
Does the human brain use transformers?
- exe34 3y ago
  No, but they can both implement a language virtual machine which appears to be able to produce intelligent behaviour with unknown bounds.
- mirekrusin 3y ago
  Yes, through ie. services like openai's chatgpt.
- dartos 3y ago
  What?
- __loam 3y ago
  Anyone telling you it does is a fraud.
  - utopcell 3y ago
    Here [1] are some "frauds" from Stanford University, Oxford University and University College London telling you exactly that. From their abstract: ``One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably place and grid cells. Furthermore, we show that this result is no surprise since it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the neuroscience version.`` [1] https://arxiv.org/abs/2112.04035 https://arxiv.org/abs/2112.04035
    - __loam 3y ago
      Making the claim that transformers are a good candidate model for certain neural pathways is a pretty different claim than saying the brain is literally using transformers.
bloopernova 3y ago
Do LLMs use neural nets? If so, what makes up the "neuron"? i.e. Is there a code structure that underlies the neuron, or is it "just" fancy math?
- theonlybutlet 3y ago
  Yes to both, the "neuron" would basically be a weighted parameter. A parameter is an expression, it's a mathematical representation of a token and it's probabilistic weighting (theyre translated from input or to output token lists entering and exiting the model). Usually tokens are pre-set small groups of character combinations like "if " or "cha" that make up a word/sentence. The recorded path your value takes down the chain of probabilities would be the "neural pathway" within the wider "neural network". Someone please correct me if I'm wrong or my terminology is wrong.
  - pmayrgundter 3y ago
    This is all true in a neutral net, but Transformers aren't Neural Nets in the traditional sense. I was under that impression originally, but there's not a back propagation or Hebbian learning here, which were the key bits of biomimicry that earned classic NNs their name. Transformers do have coefficients that are fit, but that's more broad.. could be used for any sort of regression or optimization, and not necessarily indicative of biological analogs. So I think the terms "learned model" of "weights" are malapropisms for Transformers, carried over from deep nets because of structural similarities, like many layers, and the development workflow. The functional units in Transformer's layers have lost their orginal biological inspiration and functional analog. The core function in Transformers is more like autoencoding/decoding (concepts from info theory) and model/grammar-free translation, with a unique attention based optimization. Transformers were developed for translation. The magic is smth like "attending" to important parts of the translation inputs&outputs as tokens are generated, maybe as a kind of deviation on pure autoencoding, due to the bias from the .. learned model :) See I can't even escape it. Attention as a powerful systemic optimization is the actual closer bit of neuro/bio-insporation here.. but more from Cog Psych than micro/neuro anatomy. Btw, not only is attention a key insight for Transformers, but it's an interesting biographical note that the lead inventor of it, Jakob Uzkereit, went on to work on a bio-AI startup after Google.
enriquto 3y ago
For a dryer, more formal and succinct approach, see "The Transformer Model in Equations" [0], by John Thickstun. The whole thing fits in a single page, using standard mathematical notation. [0] https://johnthickstun.com/docs/transformers.pdf https://johnthickstun.com/docs/transformers.pdf
- wardedVibe 3y ago
  Thank god, I've had to cobble something like this together for my own notes a couple of times trying to parse papers and was never quite sure if I was missing something.
- EnnioEvo 3y ago
  Finally, thank you so much! Was it so difficult? Isn't 7 lines of mathematical notation way better than pages of qualitative pub talking? I don't really understand these ML researchers, it always looks like they have never studied mathematics at all.
dogline 3y ago
Six paragraphs in, and I already have questions. > Hello -> [1,2,3,4] World -> [2,3,4,5] The vectors are random, but they look like they have a pattern here. Does the 2 in both vector mean something? Or, is it the entire set that makes it unique?
- dan-robertson 3y ago
  The number reuse is just the author being a bit lazy. You could estimate how similar these vectors are by seeing if they point in similar directions or by calculating the angle between them. Here they are about 60° apart and somewhat the same direction, but a lot of this is that the author didn’t want to put in any negative numbers in the example so vectors end up being a bit more similar than they would be really. That the numbers are reused isn’t meaningful here: a 1 in the first position is quite unrelated to a 1 in the second (as no convolutions are done over this vector)
  - dogline 3y ago
    Thank you. I guess I need to back up. This is a vector, not just an identifier, and direction and angle seem important. I need to look up how the encoding is normally done, since this isn't obvious if you haven't worked in this domain before.
    - kevindamm 3y ago
      The encoding is typically learned, and if possible is part of the ANN so that it can be adjusted along with the other parameters. A good place to start on that topic is the word2vec paper.
- smaddox 3y ago
  That isn't a very good example. The vectors for each token are randomly initialized with each element taken from the normal distribution. After training, similar words will have some cosine similarity, but almost never as much cosine similarity as [1,2,3,4] and [2,3,4,5].
adamnemecek 3y ago
It’s a renormalization process. It can be modelled as a convolution in a Hopf algebra.
- chpatrick 3y ago
  It's simple, monads are just monoids in the category of endofunctors...
snaxsnaxsnax 3y ago
Hmmm, yes, I know some of these words.
naitgacem 3y ago
Reading the title I thought this was about electrical transformers :p Although this is HN but my background is still stronger. And by the way, is it worth it to invest time to get some idea about this whole AI field? I'm from a compE background
- dataking 3y ago
  > is it worth it to invest time to get some idea about this whole AI field? I'm from a compE background Might be worth thinking about how it will specifically affect your field of expertise. Jensen Huang says your job won't be taken over by an AI but by a human using an AI.
godelski 3y ago
I love and hate these. I love them because they do give another resource at explaining models such as transformers and I think this one is pretty well done (note: you really need to do something about the equation in 4.2...) First, the critique is coming from love. Great work, so I don't want it to be taken as I'm saying anything it isn't. Why I hate these is that they are labeled as "math behind" but I think this is not quite fitting. This is the opposite of the complaint I made about the Introduction to DL post the other day[0]. The issue isn't that there isn't math, but contextually it is being labeled as a mathematical approach but I'm not seeing anything that distinguishes it as deeper than what you'd get from Karpathy's videos or the Annotated Transformer (I like this more than illustrated). There's nothing wrong with that, but just think it might mislead people, especially as there is a serious lack of places to find a much deeper mathematical explanation behind architectures and the naming makes it harder to find for those that are looking for that, because they'll find these posts. Simply, complaint is about framing. To be clear, the complaint is just about the subtitle, because the article is good and a useful resource for people seeking to learn attention and transformers. But let me try to clarify some of what would I personally (welcome to disagree, it is an opinion) more accurately be representative of " demystifying all the math behind them": - I would include a much deeper discussion of both embedding and positional embedding. The former you should at minimum be discussion how it is created and discussing the dequantization. This post may give a reader the impression that this is not taking place (there is ambiguity between distinction of embedding vs tokenization and embedding, this looks to just briefly mention tokenization. I specifically think a novice might take away that the dequantization is happening due to the positional encoding, and not in the embedding). The tokenization and embedding is a vastly underappreciated and incredibly important aspect of making discrete models work (not just LLMs or LMs. Principle is more general). - Same goes for the positional embedding which I have only in a handful of cases seen discussed and taken rather matter of factly. For a mathematical explanation you do need to explain the idea behind generating unique signals for each position, explain why we need a a high frequency, and it is worth mentioning how this can be learnable (often with similar results, which is why most don't bother), and other forms like rotational. The principle is far more general than even a Fourier Series (unmentioned!). The continuous aspect also matters a lot here, and we (often) don't want discritized positional encoding. If this isn't explained it feels rather arbitrary, and in some ways it is but others it isn't. - The attention mechanism is vastly under-explained, though I understand why. There are many approaches to tackle this, some from graphs, some from category theory, and many others. They're all valuable pieces to the puzzle. But at minimum I think there needs to be a clear identification as to what the dot product is doing, the softmax, the scale (see softmax tempering), and why we then have the value. Their key-query-value names were not chosen at random and the database analogy is quite helpful. Maybe many don't understand the relationship of dot products and angles between vectors? But this can even get complex as we would expect values to go to 0 in high dimensions (which they kinda do if you look at the attention matricies post learning which often look highly diagonal and why you can initialize them as diagonally spiked for sometimes faster training). This would be a great place to bring up how there might be some surprising aspects to the attention mechanism considering matrices represent affine transformations of data (linear) and we might not see the non-linearity here (softmax) or understand why softmax works better than other non-linears or normalizers (try it yourself!). - There's more but I've written a wall. So I'll just say we can continue for the residuals (also see META's 3 Things Everyone Should Know About Vision Transformers, in the Deit repo), why we have pre-norm as opposed to the original post-norm (which it looks like post norm is being used!), the residuals (knot theory can help here a bit), and why we have the linear layer (similarly the unknotting discussion helps, especially quantifying why we like a 4x ratio, but isn't absolutely necessary). Idk, are people interested in these things? I know most people aren't, and there's absolutely nothing wrong with that (you can still build strong models without this knowledge, but it is definitely helpful). I do feel that we often call these things black boxes but they aren't completely opaque. They sure aren't transparent, especially through scale, but they aren't "black" either. (Allen-Zhu & Li's Physics of LLMs is a great resource btw and I'd love if other users posted/referenced more things they liked. I purposefully didn't link btw) So, I do like the post, and I think it has good value (and certainly there is always value in teaching to learn!), but I disagree with the HN title and post's subtitle. [0] https://news.ycombinator.com/item?id=38834244 https://news.ycombinator.com/item?id=38834244
remexre 3y ago
Should the line Z_encoder_decoder = layer_norm(Z_encoder_decoder + Z) in Decoder step 7 instead be Z_encoder_decoder = layer_norm(Z_encoder_decoder + Z_self_attention) ? Also, is layer_norm missing in Decoder step 8...
bsenftner 3y ago
Hard to understand when concepts are used without definition or introduction. The Encoder section just begins without any description of what it is or where is sets in an overall process. I grasp what the author is trying to do, but the post misses basic essay structures such as introducing ideas and explaining them before using them, rending the entire post confusing if one is not already a student and half understands the topic before reading.
Luechkt 3y ago
I knew there was more than meets the eye.
heisenburgzero 3y ago
Not completely related. Does anyone know where I can find articles / papers that discuss why transformers, while acting as merely "next token predictor" can handle questions with: 1. Unknown words (or subwords/tokens) that are not seen in the training dataset. Example: Create a table with "sdsfs_ff", "fsdf_value" as columns in pandas. 2. Create examples(unseen in training dataset) and tell the LLM to provide similar output. I have a feeling it should be a common question, but I just can't find the keyword to search. PS. If anyone has any links with thoroughly discussion about positional embedding, that would be great. I never got a satisfying answer about the usage of sine / cosine and (multiplication vs addition)
- treyd 3y ago
  If I had to guess, single characters are able to be encoded as tokens, but there's more "bandwidth" in the model being dedicated to handling them and there's less semantic meaning encoded in them "natively" compared to tokens for concrete words. If it decides to, it can recreate unknown sequences by copying over the tokens for the single letters or create them if it makes sense.
  - heisenburgzero 3y ago
    I think some earlier NLP applications have something called "Unknown token", which they will replace any unseen word. But for recent implementations, I don't think they are being used anymore. It still baffles me why such stochastic parrot / next token predictor, will recognize these "Unseen combinations of tokens" and reuse them in response.
    - stevenhuang 3y ago
      Everything falls into place once you understand that LLMs are indeed learning hierarchical concepts inherent in the structured data it has been trained on. These concepts exist in a high dimensional latent space. Within this space is the concept of nonsense/gibberish/placeholder, which your sequence of unseen tokens map to. Then it combines this with the concept of SQL tables, resulting in hopefully the intended answer.
dingclancy 3y ago
I am loving the Quarto website. I see more Python users using Quarto for publishing.
jongjong 3y ago
As someone who has written an ANN from scratch and hasn't used TensorFlow, I still find this description confusing. I asked ChatGPT to explain how to modify a basic ANN to implement self-attention without using the terms Matrix or Vector and it gave me a really simple explanation. Though I haven't tried to implement it yet. I prefer to think of everything in terms of nodes, weights and layers. Matrices and vectors just makes it harder to relate to what's happening in the ANN. The way I'm used to writing ANNs, each input node is a scalar but the feed forward algorithm looks like vector-matrix multiplication since you multiply all the input nodes by the weights then sum them up... Anyway, I feel like I'm approaching these descriptions with the wrong mindset. Maybe I lack the necessary background.
sam16180 3y ago
Is there an error in the positional encoding example? For example when calculating PE(1, 3), I'd expect i = 1 as 3 = 2 * 1 + 1 So for “World” PE(1, 0) = sin(1 / 10000^(2*0 / 4)) = sin(1 / 10000^0) = sin(1) ≈ 0.84 PE(1, 1) = cos(1 / 10000^(2*0 / 4)) = cos(1 / 10000^0) = cos(1) ≈ 0.54 PE(1, 2) = sin(1 / 10000^(2*1 / 4)) = sin(1 / 10000^.5) ≈ 0.01 PE(1, 3) = cos(1 / 10000^(2*1 / 4)) = cos(1 / 10000^.5) ≈ 1 I also wondered if these formulae were devised with 1-based indexing in mind (though I guess for larger dimensions it doesn't make much difference), as the paper states > The wavelengths form a geometric progression from 2π to 10000 · 2π That led me to this chain of PRs - https://github.com/tensorflow/tensor2tensor/pull/177 https://github.com/tensorflow/tensor2tensor/pull/177 - turns out the original code was actually quite different to that stated in the paper. I guess slight variations in how you calculate this encoding doesn't affect things too much?
alexmolas 3y ago
> Uh oh! We’re getting NaNs! It seems our values are too high, and when being passed to the next encoder, they end up being too high and exploding! This is called gradient explosion. As far as I understand this is wrong. You're not computing gradients at any point, so this is no gradient explosion. I believe the problem is with the implementation of softmax, here [0] you have an explanation of how to implement a numerically stable softmax. [0]: https://jaykmody.com/blog/stable-softmax/ https://jaykmody.com/blog/stable-softmax/
- osanseviero 3y ago
  Yes, you're correct. I tried to connect a common training problem (gradient explosion and vanishing gradient) with the issue of softmax being sensitive to large values. I agree it's misleading/inaccurate, so will rewrite that part. That said, the whole neural network will be sensible to large values, so it won't be fixed by a numerically stable softmax. The normalization is a key aspect for the network to work.
MrVandemar 3y ago
Definitely more than meets the eye.

> The complexity comes from the number of steps and the number of parameters. Yes, it seems like a transformer model simple enough for us to understand isn't able to do anything interesting, and a transformer complex enough to do something interesting is too complex for us to understand. I would love to study something in the middle, a model that is both simple enough to understand and complex enough to do something interesting.

PeterisP 3y ago
I would assume that the boundaries of those ranges are such that the middle in between those extremes is something that is already too complex for a human to properly understand while still too small to be able to do anything interesting.
calebkaiser 3y ago
You might be interested, if you aren't already familiar, in some of the work going on in the mechanistic interpretability field. Neel Nanda has a lot of approachable work on the topic: https://www.neelnanda.io/mechanistic-interpretability https://www.neelnanda.io/mechanistic-interpretability
- leereeves 3y ago
  I was not familiar with it, and that does look fascinating, thank you. If anyone else is interested, this guide "Concrete Steps to Get Started in Transformer Mechanistic Interpretability" on his site looks like a great place to start: https://www.neelnanda.io/mechanistic-interpretability/getting-started https://www.neelnanda.io/mechanistic-interpretability/gettin...

quickthrower2 3y ago

Transformer tutorials might be the new monad tutorial. A hard concept to get, but one you need to struggle with (and practice some examples) to understand. So a bit like much of computer science :-).

hdhfjkrkrme 3y ago
The moment you understand the Transformer you become incapable of explaining it.
amelius 3y ago
Waiting for a blogpost titled "You could have invented transformers".
csdvrx 3y ago
> Transformer tutorials might be the new monad tutorial. A hard concept to get, A hard concept? But a monad is just a monoid in the category of endofunctors, so what's the problem?

nemo8551 3y ago

There I was all excited to show off some of my electrical chops on HN. Not today.

rzzzt 3y ago
Does mystified math lie beyond behind how the ratio of input and output voltages is equal to the ratio of the primary and secondary windings? Can it be derived from Maxwell's equations? Off to a search...
- amelius 3y ago
  I bet that an LLM (which uses transformers) can explain those aspects of a transformer to you.
  - wardedVibe 3y ago
    Or just read Wikipedia, which I'm sure it will crib from poorly... https://en.wikipedia.org/wiki/Transformer https://en.wikipedia.org/wiki/Transformer
  - nemo8551 3y ago
    I seen an LLM, or maybe another variant of “AI” [0] a while back that could aid design of electronic circuits by having a pool of data sheets added for referencing. As you were querying specs for a board at component level it could give you a schematic, I think, with citations to the actual data sheets. I suppose the same scale up could be used for systems that needed a varying number of specific power supplies. [0] https://www.flux.ai/p https://www.flux.ai/p
    - amelius 3y ago
      Actually, in my experience chatgpt knows quite a lot about electronic components so I wouldn't be surprised if it was fed datasheets.
      - 3y ago

ActorNightly 3y ago

The whole "mystery" of transformer is that instead of a linear sequence of static weights times values in each layer, you now have 3 different matrices that are obtained from the same input through multiplication of learned weights, and then you just multiply the matrices together. I.e more parallelism which works out nice, but very restrictive since the attention formula is static. We arent going to see more progress until we have a way to generalize the compute graph as a learnable parameter. I dunno if this is even possible in the traditional sense of gradients due to chaotic effects (i.e small changes reflect big shifts in performance), it may have to be some form of genetic algorithm or pso that happens under the hood.

ex3ndr 3y ago
This is basically this - it can learn ignore some paths, and amplify something more important, then you can just cut this paths without sensible loss of quality. The problem is that you are not going to win anything from this - non-matrix multiplication would be slower or the same.
- ActorNightly 3y ago
  The issue is that you are thinking of this in terms of information compression, which is what LLMs are. Im more concerned with an LLM having the ability to be trained to the point where a subset of the graph represents all the nand gates necessary for a cpu and ram, so when you ask it questions it can actually run code to compute them accurately instead of offering a statistical best guess, i.e decompression after lossy compression.
  - exe34 3y ago
    Just give it a computer? Even a virtual machine. It can output assembly code or high level code that gets compiled.
    - ActorNightly 3y ago
      The issue is not having access to the cpu, the issue is that the model being able to be trained in such a way that it has representative structures for applicable problem solving. Furthermore, the structures itself should Philosophically, you can start ad hoc-ing functionalities on top of LLMs and expect major progress. Sure, you can make them better, but you will never get to the state where AI is massively useful. For example, lets say you gather a whole bunch of experts in respective fields, and you give them a task to put together a detailed plan on how to build a flying car. You will have people doing design, doing simulations, researching material sourcing, creating CNC programs for manufacturing parts, sourcing tools and equipment, writing software, e.t.c. And when executing this plan, they would be open to feedback for anything missed, and can advise on how to proceed. The AI with above capability should be able to go out on the internet, gather respective data, run any soft of algorithms it needs to run, and perhaps after a month of number crunching on a cloud rented TPU rack produce step by step plan with costs on how to do all of that. And it would be better than those experts because it should be able to create a much higher fidelity simulations to account for things like vibration and predict if some connector if going to wobble loose .

brcmthrowaway 3y ago

Does the human brain use transformers?

exe34 3y ago
No, but they can both implement a language virtual machine which appears to be able to produce intelligent behaviour with unknown bounds.
mirekrusin 3y ago
Yes, through ie. services like openai's chatgpt.
dartos 3y ago
What?
__loam 3y ago
Anyone telling you it does is a fraud.
- utopcell 3y ago
  Here [1] are some "frauds" from Stanford University, Oxford University and University College London telling you exactly that. From their abstract: ``One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably place and grid cells. Furthermore, we show that this result is no surprise since it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the neuroscience version.`` [1] https://arxiv.org/abs/2112.04035 https://arxiv.org/abs/2112.04035
  - __loam 3y ago
    Making the claim that transformers are a good candidate model for certain neural pathways is a pretty different claim than saying the brain is literally using transformers.

bloopernova 3y ago

Do LLMs use neural nets? If so, what makes up the "neuron"? i.e. Is there a code structure that underlies the neuron, or is it "just" fancy math?

theonlybutlet 3y ago
Yes to both, the "neuron" would basically be a weighted parameter. A parameter is an expression, it's a mathematical representation of a token and it's probabilistic weighting (theyre translated from input or to output token lists entering and exiting the model). Usually tokens are pre-set small groups of character combinations like "if " or "cha" that make up a word/sentence. The recorded path your value takes down the chain of probabilities would be the "neural pathway" within the wider "neural network". Someone please correct me if I'm wrong or my terminology is wrong.
- pmayrgundter 3y ago
  This is all true in a neutral net, but Transformers aren't Neural Nets in the traditional sense. I was under that impression originally, but there's not a back propagation or Hebbian learning here, which were the key bits of biomimicry that earned classic NNs their name. Transformers do have coefficients that are fit, but that's more broad.. could be used for any sort of regression or optimization, and not necessarily indicative of biological analogs. So I think the terms "learned model" of "weights" are malapropisms for Transformers, carried over from deep nets because of structural similarities, like many layers, and the development workflow. The functional units in Transformer's layers have lost their orginal biological inspiration and functional analog. The core function in Transformers is more like autoencoding/decoding (concepts from info theory) and model/grammar-free translation, with a unique attention based optimization. Transformers were developed for translation. The magic is smth like "attending" to important parts of the translation inputs&outputs as tokens are generated, maybe as a kind of deviation on pure autoencoding, due to the bias from the .. learned model :) See I can't even escape it. Attention as a powerful systemic optimization is the actual closer bit of neuro/bio-insporation here.. but more from Cog Psych than micro/neuro anatomy. Btw, not only is attention a key insight for Transformers, but it's an interesting biographical note that the lead inventor of it, Jakob Uzkereit, went on to work on a bio-AI startup after Google.

enriquto 3y ago

For a dryer, more formal and succinct approach, see "The Transformer Model in Equations" [0], by John Thickstun. The whole thing fits in a single page, using standard mathematical notation. [0] https://johnthickstun.com/docs/transformers.pdf https://johnthickstun.com/docs/transformers.pdf

wardedVibe 3y ago
Thank god, I've had to cobble something like this together for my own notes a couple of times trying to parse papers and was never quite sure if I was missing something.
EnnioEvo 3y ago
Finally, thank you so much! Was it so difficult? Isn't 7 lines of mathematical notation way better than pages of qualitative pub talking? I don't really understand these ML researchers, it always looks like they have never studied mathematics at all.

dogline 3y ago

Six paragraphs in, and I already have questions. > Hello -> [1,2,3,4] World -> [2,3,4,5] The vectors are random, but they look like they have a pattern here. Does the 2 in both vector mean something? Or, is it the entire set that makes it unique?

dan-robertson 3y ago
The number reuse is just the author being a bit lazy. You could estimate how similar these vectors are by seeing if they point in similar directions or by calculating the angle between them. Here they are about 60° apart and somewhat the same direction, but a lot of this is that the author didn’t want to put in any negative numbers in the example so vectors end up being a bit more similar than they would be really. That the numbers are reused isn’t meaningful here: a 1 in the first position is quite unrelated to a 1 in the second (as no convolutions are done over this vector)
- dogline 3y ago
  Thank you. I guess I need to back up. This is a vector, not just an identifier, and direction and angle seem important. I need to look up how the encoding is normally done, since this isn't obvious if you haven't worked in this domain before.
  - kevindamm 3y ago
    The encoding is typically learned, and if possible is part of the ANN so that it can be adjusted along with the other parameters. A good place to start on that topic is the word2vec paper.
smaddox 3y ago
That isn't a very good example. The vectors for each token are randomly initialized with each element taken from the normal distribution. After training, similar words will have some cosine similarity, but almost never as much cosine similarity as [1,2,3,4] and [2,3,4,5].

adamnemecek 3y ago

It’s a renormalization process. It can be modelled as a convolution in a Hopf algebra.

chpatrick 3y ago
It's simple, monads are just monoids in the category of endofunctors...

snaxsnaxsnax 3y ago

Hmmm, yes, I know some of these words.

naitgacem 3y ago

Reading the title I thought this was about electrical transformers :p Although this is HN but my background is still stronger. And by the way, is it worth it to invest time to get some idea about this whole AI field? I'm from a compE background

dataking 3y ago
> is it worth it to invest time to get some idea about this whole AI field? I'm from a compE background Might be worth thinking about how it will specifically affect your field of expertise. Jensen Huang says your job won't be taken over by an AI but by a human using an AI.

godelski 3y ago

I love and hate these. I love them because they do give another resource at explaining models such as transformers and I think this one is pretty well done (note: you really need to do something about the equation in 4.2...) First, the critique is coming from love. Great work, so I don't want it to be taken as I'm saying anything it isn't. Why I hate these is that they are labeled as "math behind" but I think this is not quite fitting. This is the opposite of the complaint I made about the Introduction to DL post the other day[0]. The issue isn't that there isn't math, but contextually it is being labeled as a mathematical approach but I'm not seeing anything that distinguishes it as deeper than what you'd get from Karpathy's videos or the Annotated Transformer (I like this more than illustrated). There's nothing wrong with that, but just think it might mislead people, especially as there is a serious lack of places to find a much deeper mathematical explanation behind architectures and the naming makes it harder to find for those that are looking for that, because they'll find these posts. Simply, complaint is about framing. To be clear, the complaint is just about the subtitle, because the article is good and a useful resource for people seeking to learn attention and transformers. But let me try to clarify some of what would I personally (welcome to disagree, it is an opinion) more accurately be representative of " demystifying all the math behind them": - I would include a much deeper discussion of both embedding and positional embedding. The former you should at minimum be discussion how it is created and discussing the dequantization. This post may give a reader the impression that this is not taking place (there is ambiguity between distinction of embedding vs tokenization and embedding, this looks to just briefly mention tokenization. I specifically think a novice might take away that the dequantization is happening due to the positional encoding, and not in the embedding). The tokenization and embedding is a vastly underappreciated and incredibly important aspect of making discrete models work (not just LLMs or LMs. Principle is more general). - Same goes for the positional embedding which I have only in a handful of cases seen discussed and taken rather matter of factly. For a mathematical explanation you do need to explain the idea behind generating unique signals for each position, explain why we need a a high frequency, and it is worth mentioning how this can be learnable (often with similar results, which is why most don't bother), and other forms like rotational. The principle is far more general than even a Fourier Series (unmentioned!). The continuous aspect also matters a lot here, and we (often) don't want discritized positional encoding. If this isn't explained it feels rather arbitrary, and in some ways it is but others it isn't. - The attention mechanism is vastly under-explained, though I understand why. There are many approaches to tackle this, some from graphs, some from category theory, and many others. They're all valuable pieces to the puzzle. But at minimum I think there needs to be a clear identification as to what the dot product is doing, the softmax, the scale (see softmax tempering), and why we then have the value. Their key-query-value names were not chosen at random and the database analogy is quite helpful. Maybe many don't understand the relationship of dot products and angles between vectors? But this can even get complex as we would expect values to go to 0 in high dimensions (which they kinda do if you look at the attention matricies post learning which often look highly diagonal and why you can initialize them as diagonally spiked for sometimes faster training). This would be a great place to bring up how there might be some surprising aspects to the attention mechanism considering matrices represent affine transformations of data (linear) and we might not see the non-linearity here (softmax) or understand why softmax works better than other non-linears or normalizers (try it yourself!). - There's more but I've written a wall. So I'll just say we can continue for the residuals (also see META's 3 Things Everyone Should Know About Vision Transformers, in the Deit repo), why we have pre-norm as opposed to the original post-norm (which it looks like post norm is being used!), the residuals (knot theory can help here a bit), and why we have the linear layer (similarly the unknotting discussion helps, especially quantifying why we like a 4x ratio, but isn't absolutely necessary). Idk, are people interested in these things? I know most people aren't, and there's absolutely nothing wrong with that (you can still build strong models without this knowledge, but it is definitely helpful). I do feel that we often call these things black boxes but they aren't completely opaque. They sure aren't transparent, especially through scale, but they aren't "black" either. (Allen-Zhu & Li's Physics of LLMs is a great resource btw and I'd love if other users posted/referenced more things they liked. I purposefully didn't link btw) So, I do like the post, and I think it has good value (and certainly there is always value in teaching to learn!), but I disagree with the HN title and post's subtitle. [0] https://news.ycombinator.com/item?id=38834244 https://news.ycombinator.com/item?id=38834244

remexre 3y ago

Should the line Z_encoder_decoder = layer_norm(Z_encoder_decoder + Z) in Decoder step 7 instead be Z_encoder_decoder = layer_norm(Z_encoder_decoder + Z_self_attention) ? Also, is layer_norm missing in Decoder step 8...

bsenftner 3y ago

Hard to understand when concepts are used without definition or introduction. The Encoder section just begins without any description of what it is or where is sets in an overall process. I grasp what the author is trying to do, but the post misses basic essay structures such as introducing ideas and explaining them before using them, rending the entire post confusing if one is not already a student and half understands the topic before reading.

Luechkt 3y ago

I knew there was more than meets the eye.

heisenburgzero 3y ago

Not completely related. Does anyone know where I can find articles / papers that discuss why transformers, while acting as merely "next token predictor" can handle questions with: 1. Unknown words (or subwords/tokens) that are not seen in the training dataset. Example: Create a table with "sdsfs_ff", "fsdf_value" as columns in pandas. 2. Create examples(unseen in training dataset) and tell the LLM to provide similar output. I have a feeling it should be a common question, but I just can't find the keyword to search. PS. If anyone has any links with thoroughly discussion about positional embedding, that would be great. I never got a satisfying answer about the usage of sine / cosine and (multiplication vs addition)

treyd 3y ago
If I had to guess, single characters are able to be encoded as tokens, but there's more "bandwidth" in the model being dedicated to handling them and there's less semantic meaning encoded in them "natively" compared to tokens for concrete words. If it decides to, it can recreate unknown sequences by copying over the tokens for the single letters or create them if it makes sense.
- heisenburgzero 3y ago
  I think some earlier NLP applications have something called "Unknown token", which they will replace any unseen word. But for recent implementations, I don't think they are being used anymore. It still baffles me why such stochastic parrot / next token predictor, will recognize these "Unseen combinations of tokens" and reuse them in response.
  - stevenhuang 3y ago
    Everything falls into place once you understand that LLMs are indeed learning hierarchical concepts inherent in the structured data it has been trained on. These concepts exist in a high dimensional latent space. Within this space is the concept of nonsense/gibberish/placeholder, which your sequence of unseen tokens map to. Then it combines this with the concept of SQL tables, resulting in hopefully the intended answer.

dingclancy 3y ago

I am loving the Quarto website. I see more Python users using Quarto for publishing.

jongjong 3y ago

As someone who has written an ANN from scratch and hasn't used TensorFlow, I still find this description confusing. I asked ChatGPT to explain how to modify a basic ANN to implement self-attention without using the terms Matrix or Vector and it gave me a really simple explanation. Though I haven't tried to implement it yet. I prefer to think of everything in terms of nodes, weights and layers. Matrices and vectors just makes it harder to relate to what's happening in the ANN. The way I'm used to writing ANNs, each input node is a scalar but the feed forward algorithm looks like vector-matrix multiplication since you multiply all the input nodes by the weights then sum them up... Anyway, I feel like I'm approaching these descriptions with the wrong mindset. Maybe I lack the necessary background.

sam16180 3y ago

Is there an error in the positional encoding example? For example when calculating PE(1, 3), I'd expect i = 1 as 3 = 2 * 1 + 1 So for “World” PE(1, 0) = sin(1 / 10000^(2*0 / 4)) = sin(1 / 10000^0) = sin(1) ≈ 0.84 PE(1, 1) = cos(1 / 10000^(2*0 / 4)) = cos(1 / 10000^0) = cos(1) ≈ 0.54 PE(1, 2) = sin(1 / 10000^(2*1 / 4)) = sin(1 / 10000^.5) ≈ 0.01 PE(1, 3) = cos(1 / 10000^(2*1 / 4)) = cos(1 / 10000^.5) ≈ 1 I also wondered if these formulae were devised with 1-based indexing in mind (though I guess for larger dimensions it doesn't make much difference), as the paper states > The wavelengths form a geometric progression from 2π to 10000 · 2π That led me to this chain of PRs - https://github.com/tensorflow/tensor2tensor/pull/177 https://github.com/tensorflow/tensor2tensor/pull/177 - turns out the original code was actually quite different to that stated in the paper. I guess slight variations in how you calculate this encoding doesn't affect things too much?

alexmolas 3y ago

> Uh oh! We’re getting NaNs! It seems our values are too high, and when being passed to the next encoder, they end up being too high and exploding! This is called gradient explosion. As far as I understand this is wrong. You're not computing gradients at any point, so this is no gradient explosion. I believe the problem is with the implementation of softmax, here [0] you have an explanation of how to implement a numerically stable softmax. [0]: https://jaykmody.com/blog/stable-softmax/ https://jaykmody.com/blog/stable-softmax/

osanseviero 3y ago
Yes, you're correct. I tried to connect a common training problem (gradient explosion and vanishing gradient) with the issue of softmax being sensitive to large values. I agree it's misleading/inaccurate, so will rewrite that part. That said, the whole neural network will be sensible to large values, so it won't be fixed by a numerically stable softmax. The normalization is a key aspect for the network to work.

MrVandemar 3y ago

Definitely more than meets the eye.