4 ms·

But what is a GPT? Visual intro to Transformers [video]

473 points by huhhuh 2y ago

lucidrains 2y ago
I can't think of anyone better to teach attention mechanism to the masses. This is a dream come true
- acchow 2y ago
  Incredible. This 3B1B series was started 6 years ago and keeps going today with chapter 5. If you haven't seen the first few chapters, I cannot recommend enough.
  - user_7832 2y ago
    Would you be able to compare them to Andrew Ng's course?
    - sk11001 2y ago
      They're not really comparable - if you're wondering if you should do one or the other, you should do both.
    - ctrw 2y ago
      The way you compare a technical drawing of a steam engine to The Fighting Temeraire oil painting.
    - abraxas 2y ago
      I personally preferred Andrej Karpathy's CS231n taught by him and his private videos about neural nets in general and transformers in particular. He has a youtube vid where he builds one from scratch in Python! 3BlueOneBrown videos are a great complement to Karpathy's lectures to aid in visualising what is going on.
    - mFixman 2y ago
      IMO the style, formatting, and animations in 3B1B videos is what Coursera courses should have been about in the first place. Andrew Ng's course doesn't use video effectively at all: half of each class is Andrew talking to the camera, while the other half is him slowly writing things down with a mouse. There's a reason why a lot of people recommend watching at 1.5x speed. Online classes are online classes. If they try to make copy in-person classes, like most Coursera courses do, they will keep all of the weaknesses of online classes without any of its strengths.
    - spmurrayzzz 2y ago
      IMO I think the 3Blue1Brown video is a good place to start to build intuitions about how things work generally if you're new, and Andrew Ng's courses will help you dig into more detail, experiment, and implement things to build on those intuitions.
yinser 2y ago
What an unbelievable salve for all the April Fool's content. Pipe this directly into my veins.
user_7832 2y ago
I've just started this video, but already have a question if anyone's familiar with GPT workings - I thought that these models chose the next word based on what's most likely. But if they choose based on "one of the likely" words, could (in general) that not lead to a situation where the list of predictions for the next word are much less likely? Running possibilities of "two words together", then, would be more beneficial if computationally possible (and so on for 3, 4 and n words). Does this exist? (I realize that choosing the most likely word wouldn't necessarily solve the issue, but choosing the most likely phrase possibly might.) Edit, post seeing the video and comments: it's beam search, along with temperature to control these things.
- mvsin 2y ago
  Something like this does exist, production systems rarely use greedy search but have more holistic search algorithms. An example is Beam Search:https://www.width.ai/post/what-is-beam-search https://www.width.ai/post/what-is-beam-search Essentially we keep a window of probabilities of predicted tokens to improve the final quality of output.
  - user_7832 2y ago
    Thanks, that's exactly what I was looking for! Any idea if it's possible to use beam search on local models like mistral? It sounds like the choice of beam search vs say top-p or top-k should be in the software and not embedded, right?
    - yunohn 2y ago
      This is actually a great question for which I found an interesting attempt: https://andys.page/posts/llm_sampling_strategies/ https://andys.page/posts/llm_sampling_strategies/ (No affiliation)
    - activatedgeek 2y ago
      If you use HuggingFace models, then a few simpler decoding algorithms are already implemented for `generate` method of all supported models. Here is a blog post that describes it: https://huggingface.co/blog/how-to-generate https://huggingface.co/blog/how-to-generate. I will warn you though that beam search is typically what you do NOT want. Beam search approximately optimizes for the "highest likely sequence at the token level." This is rarely what you need in practice with open-ended generations (e.g. a question-answering chat bot). In practice, you need "highest likely semantic sequence," which is much harder problem. Of course, various approximations for semantic alignment are currently in the literature, but still a wide open problem.
arlcode 2y ago
If you liked that, Andrej karpathy has a few interesting videos on his channels explaining Neural Networks and their inner workings which are aimed at people who know how to program.
- jtonz 2y ago
  As a reasonably experienced programmer that has watched Andrej's videos the one thing I would recommend is that they not be used as a starting point to learn neural networks but as a reinforcement or enhancement method once you know the fundamentals. I was ignorant enough to try and jump straight in to his videos and despite him recommending I watch his preceeding videos I incorrectly assumed I could figure it out as I went. There is verbiage in there that you simply must know to get the most out of it. After giving up, going away and filling in the gaps though some other learnings, I went back and his videos become (understandably) massively more valueable for me. I would strongly recommend anyone else wanting to learn neural networks that they learn from my mistake.
  - kovrik 2y ago
    Could you please share what other learning materials you used?
    - 6mian 2y ago
      For me 3brown1blue series: https://m.youtube.com/watch?v=aircAruvnKk https://m.youtube.com/watch?v=aircAruvnKk was an excellent introduction that made Andrej's videos understandable. Then I did 3 first chapters of fastai book, but found it too high level, while I was interested in how things works under the hood. Going through Andrej's makemore tutorials required quite a lot of time but it's definitely worth it. I used free tier of Google Colab until the last one. Pausing the video a lot after he explains what he plans to do and trying to do it by myself was a very rewarding way to learn, with a lot of "aha" moments.
__loam 2y ago
3B1B is one of the best stem educators in YouTube.
lxe 2y ago
Can't wait for the next videos. I think I'll finally be able to internalize and understand how these things work.
throwawayk7h 2y ago
The next token is taken by sampling the logits in the final column after unembedding. But isn't that just the last token again? Or is the matrix resized to N+1 at some step?
- HarHarVeryFunny 2y ago
  There is an end-of-sequence token appended to the input sequence, and this is what is transformed into the predicted next token.
Terr_ 2y ago
Also relevant would be this interactive visualization: https://bbycroft.net/llm https://bbycroft.net/llm Prior discussion: https://news.ycombinator.com/item?id=38505211 https://news.ycombinator.com/item?id=38505211
deleted 2y ago
[deleted]
lovestaco 2y ago
Good channel, I just get most of what he says.
RecycledEle 2y ago
Great video. Thank you for sharing.

473 points by huhhuh 2y ago

lucidrains 2y ago

I can't think of anyone better to teach attention mechanism to the masses. This is a dream come true

acchow 2y ago
Incredible. This 3B1B series was started 6 years ago and keeps going today with chapter 5. If you haven't seen the first few chapters, I cannot recommend enough.
- user_7832 2y ago
  Would you be able to compare them to Andrew Ng's course?
  - sk11001 2y ago
    They're not really comparable - if you're wondering if you should do one or the other, you should do both.
  - ctrw 2y ago
    The way you compare a technical drawing of a steam engine to The Fighting Temeraire oil painting.
  - abraxas 2y ago
    I personally preferred Andrej Karpathy's CS231n taught by him and his private videos about neural nets in general and transformers in particular. He has a youtube vid where he builds one from scratch in Python! 3BlueOneBrown videos are a great complement to Karpathy's lectures to aid in visualising what is going on.
  - mFixman 2y ago
    IMO the style, formatting, and animations in 3B1B videos is what Coursera courses should have been about in the first place. Andrew Ng's course doesn't use video effectively at all: half of each class is Andrew talking to the camera, while the other half is him slowly writing things down with a mouse. There's a reason why a lot of people recommend watching at 1.5x speed. Online classes are online classes. If they try to make copy in-person classes, like most Coursera courses do, they will keep all of the weaknesses of online classes without any of its strengths.
  - spmurrayzzz 2y ago
    IMO I think the 3Blue1Brown video is a good place to start to build intuitions about how things work generally if you're new, and Andrew Ng's courses will help you dig into more detail, experiment, and implement things to build on those intuitions.

yinser 2y ago

What an unbelievable salve for all the April Fool's content. Pipe this directly into my veins.

user_7832 2y ago

I've just started this video, but already have a question if anyone's familiar with GPT workings - I thought that these models chose the next word based on what's most likely. But if they choose based on "one of the likely" words, could (in general) that not lead to a situation where the list of predictions for the next word are much less likely? Running possibilities of "two words together", then, would be more beneficial if computationally possible (and so on for 3, 4 and n words). Does this exist? (I realize that choosing the most likely word wouldn't necessarily solve the issue, but choosing the most likely phrase possibly might.) Edit, post seeing the video and comments: it's beam search, along with temperature to control these things.

mvsin 2y ago
Something like this does exist, production systems rarely use greedy search but have more holistic search algorithms. An example is Beam Search:https://www.width.ai/post/what-is-beam-search https://www.width.ai/post/what-is-beam-search Essentially we keep a window of probabilities of predicted tokens to improve the final quality of output.
- user_7832 2y ago
  Thanks, that's exactly what I was looking for! Any idea if it's possible to use beam search on local models like mistral? It sounds like the choice of beam search vs say top-p or top-k should be in the software and not embedded, right?
  - yunohn 2y ago
    This is actually a great question for which I found an interesting attempt: https://andys.page/posts/llm_sampling_strategies/ https://andys.page/posts/llm_sampling_strategies/ (No affiliation)
  - activatedgeek 2y ago
    If you use HuggingFace models, then a few simpler decoding algorithms are already implemented for `generate` method of all supported models. Here is a blog post that describes it: https://huggingface.co/blog/how-to-generate https://huggingface.co/blog/how-to-generate. I will warn you though that beam search is typically what you do NOT want. Beam search approximately optimizes for the "highest likely sequence at the token level." This is rarely what you need in practice with open-ended generations (e.g. a question-answering chat bot). In practice, you need "highest likely semantic sequence," which is much harder problem. Of course, various approximations for semantic alignment are currently in the literature, but still a wide open problem.

arlcode 2y ago

If you liked that, Andrej karpathy has a few interesting videos on his channels explaining Neural Networks and their inner workings which are aimed at people who know how to program.

jtonz 2y ago
As a reasonably experienced programmer that has watched Andrej's videos the one thing I would recommend is that they not be used as a starting point to learn neural networks but as a reinforcement or enhancement method once you know the fundamentals. I was ignorant enough to try and jump straight in to his videos and despite him recommending I watch his preceeding videos I incorrectly assumed I could figure it out as I went. There is verbiage in there that you simply must know to get the most out of it. After giving up, going away and filling in the gaps though some other learnings, I went back and his videos become (understandably) massively more valueable for me. I would strongly recommend anyone else wanting to learn neural networks that they learn from my mistake.
- kovrik 2y ago
  Could you please share what other learning materials you used?
  - 6mian 2y ago
    For me 3brown1blue series: https://m.youtube.com/watch?v=aircAruvnKk https://m.youtube.com/watch?v=aircAruvnKk was an excellent introduction that made Andrej's videos understandable. Then I did 3 first chapters of fastai book, but found it too high level, while I was interested in how things works under the hood. Going through Andrej's makemore tutorials required quite a lot of time but it's definitely worth it. I used free tier of Google Colab until the last one. Pausing the video a lot after he explains what he plans to do and trying to do it by myself was a very rewarding way to learn, with a lot of "aha" moments.

__loam 2y ago

3B1B is one of the best stem educators in YouTube.

lxe 2y ago

Can't wait for the next videos. I think I'll finally be able to internalize and understand how these things work.

throwawayk7h 2y ago

The next token is taken by sampling the logits in the final column after unembedding. But isn't that just the last token again? Or is the matrix resized to N+1 at some step?

HarHarVeryFunny 2y ago
There is an end-of-sequence token appended to the input sequence, and this is what is transformed into the predicted next token.

Terr_ 2y ago

Also relevant would be this interactive visualization: https://bbycroft.net/llm https://bbycroft.net/llm Prior discussion: https://news.ycombinator.com/item?id=38505211 https://news.ycombinator.com/item?id=38505211

deleted 2y ago

[deleted]

lovestaco 2y ago

Good channel, I just get most of what he says.

RecycledEle 2y ago

Great video. Thank you for sharing.