7 ms·

Mamba Explained

201 points by andreyk 2y ago

programjames 2y ago
This is the best explanation I have seen for Mamba.
- spxneo 2y ago
  TLDR: Friendship ended with transformers. Now Mamba is my best friend.
bobse 2y ago
[dead]
xz18r 2y ago
I just have to say it: that image shows gunpla, i.e. Mobile Suit Gundam, not Transformers!
- throwup238 2y ago
  An official request has been made to ICANN to rescind the OP's nerd card.
andy_xor_andrew 2y ago
> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions. Lately I've been wondering... is this a problem, or a strength? It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like transformers), rather you have a state in your mind that represents the grammar and concepts, which is continuously updated as you speak (more similar to SSMs). Similarly, when you read a book, every time you read a word, you are not attending to every previous word in the book. Your model of "the book" is rather a fuzzy/approximate state that is updated with new information every time a new word appears. Right? (I'm sorry I know this is very handwavy and psuedoscientific but bear with me). Ok, so if (big if) you feel like the above is true, then to match human-type language modelling, SSMs seem more human-like than transformers. BUT... then aren't transformers strictly better in terms of accuracy? Because a transformer never "forgets" information, as long as it is within the context window, because it revisits that information every time it emits a new token. So let's say we can remove the "quadratic attention" problem of transformers with SSMs. That's a nice training/inference performance boost. But... look at where we got with "naive" attention. GPT 4, Claude 3. It's not like we're hitting a wall with quadratic attention. It's absurdly more expensive than SSMs, but GPUs certainly aren't getting slower. If all AI work stops now, and only hardware improves, it wouldn't be long until GPT4 could run on local hardware, right, provided Moore's law? /end rant, not really sure what my point was, I'm not against SSMs (they're cool) but rather I'm wondering if the SOTA will ever be SSM when attention is so damn good
- tippytippytango 2y ago
  It's a tradeoff to be managed depending on the application rather than a problem.
- maccam912 2y ago
  It depends on the task I imagine. Like writing a novel was mentioned, keeping important story lines in your memory for a long time will be necessary, or at least certainly more important than remembering what the characters were eating for lunch on page 10. But if you need to find that one loophole in a contact you probably will benefit from the perfect recall.
- spxneo 2y ago
  very good point and the sooner we can accept this difference (we access hyperdimensional entities we discover through language and math via fast and slow access and vocalize it through the alphabets we learned to read) the more "intelligence" we can unlock from AI.
- aCoreyJ 2y ago
  We're running out of the ability to make transistors smaller and closer together so beyond some major breakthrough I wouldnt expect Moore's law to continue nearly long enough to get to the point of running GPT4 on consumer hardware in the short term
  - timschmidt 2y ago
    Ah, but we've just begun stacking transistors in the third dimension.
    - jazzyjackson 2y ago
      It was never a solution, Moore's law has more than one dimension as well, not just density but heat dissipation. Can't cool down a transistor that's surrounded by transistors on all sides.
    - ctrw 2y ago
password4321 2y ago
Links to more about Mamba (selective state space models) on HN yesterday: https://news.ycombinator.com/item?id=39853958#39855430 https://news.ycombinator.com/item?id=39853958#39855430
- fisian 2y ago
  This submission has the same content as the link here (submitted to HN about a month ago): https://news.ycombinator.com/item?id=39501982 https://news.ycombinator.com/item?id=39501982 https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html
  - jimmySixDOF 2y ago
    Yes and its the same author this time published on the Gradient (the link before was to the personal blog). The Gradient by the way are amazing curators of AI news in general and have one of the better podcasts I am aware of interviewing developers in the trenches. Adding: this resurgence in Mamba in general is also due to some actual sota progress with SSM like the new AI21 lab released this week [1] and likely to see others merging different architecture layers (this is a 52B MoE with 12B params active during inference blending both Mamba and transformers) >As the first production-grade model based on Mamba architecture, Jamba achieves an unprecedented 3X throughput and fits 140K context on a single GPU. [1] https://www.ai21.com/jamba https://www.ai21.com/jamba
etbebl 2y ago
Anyone else keep seeing articles about Mamba and thinking it's about Python/Conda? It's annoying when the new cool thing picks the same name as something else you like that deserves attention.
- ragebol 2y ago
  > attention I see what you did there
- tempaccount420 2y ago
  Sounds like you need a language model to help you categorize Mamba articles into Python and non-Python articles?
Donaldwide 2y ago
[dead]
sp332 2y ago
So in an effective Mamba query the question goes at the end, after input data? I thought that the question should go at the beginning, so it can decide which information in the data is relevant.
- eropple 2y ago
  I could be wrong, as I haven't used Mamba, but it seems to remain similar to transformers in that it doesn't "decide" anything and streams tokens to follow the existing ones; attention isn't a thing in the same way, but recency does still have impact. To that end, putting context after the question makes it more likely to follow the context, not the question.
jongjong 2y ago
I find it difficult to understand certain math and science papers/articles due to ambiguous use of language. For example "all previous tokens can be passed to the current token." That seems like a poorly constructed sentence. A token is not a function and it's not an algorithm either... How can you pass tokens to a token? This type of ambiguous language in academic papers makes it hard to read... Maybe the phrase 'every token has an association with every other previously encountered token' would be better? Or every token is used to compute the token vector for each token... I don't know, all I can do is guess the meaning of the word 'passed'. They want us to infer and fill in the gaps with our own assumptions. It assumes that we are primed to think in a certain highly constrained way... For some reason a lot of academia around AI is littered with such imprecise language. They choose to use niche concepts and repurposed wording that their own small community invented rather using words and ideas that are more widely understood but which would convey the same information. Rational people who aren't directly involved in those fields who generally resist jumping to conclusions will struggle to understand what is meant because a lot of those words and ideas have different interpretations in their own fields. I studied machine learning at university and wrote ANNs from scratch and trained them and even I find the language and concepts around LLMs too ambiguous. I'd rather just ask ChatGPT. One thing that bothers me is that the community has moved away from relating concepts to neurons, interconnections, input layers, hidden layers and output layers. Instead, they jump straight into vectors and matrices... Pretending as though there is only one way to map those calculations to neurons and weights. But in fact, this abstraction has many possible interpretations. You could have fully connected layers or partially connected layers... Maybe you need a transformer only in front of the input layer or between every layer... So many possibilities. The entire article means little if considered in isolation outside of the context of current configurations of various popular frameworks and tools.
- king_magic 2y ago
  that's not what it says in the article. it actually says "information from all previous tokens can be passed to the current token". that statement is meaningfully different from "all previous tokens can be passed to the current token". and both really makes sense if you understand attention mechanisms.
ein0p 2y ago
This is more human like, and people will complain that it doesn’t have photographic memory. That is, it’s not superhuman in that regard. But there are many tasks where superhuman recall is not required. We know this because those tasks are currently performed by humans.

Mamba Explained

201 points by andreyk 2y ago

programjames 2y ago

This is the best explanation I have seen for Mamba.

spxneo 2y ago
TLDR: Friendship ended with transformers. Now Mamba is my best friend.

bobse 2y ago

[dead]

xz18r 2y ago

I just have to say it: that image shows gunpla, i.e. Mobile Suit Gundam, not Transformers!

throwup238 2y ago
An official request has been made to ICANN to rescind the OP's nerd card.

andy_xor_andrew 2y ago

> But Transformers have one core problem. In a transformer, every token can look back at every previous token when making predictions. Lately I've been wondering... is this a problem, or a strength? It might be a fallacy to compare how LLMs "think" with how humans think. But humor me for a second. When you are speaking, each time you emit a word, you are not attending to every previous word in your sentence (like transformers), rather you have a state in your mind that represents the grammar and concepts, which is continuously updated as you speak (more similar to SSMs). Similarly, when you read a book, every time you read a word, you are not attending to every previous word in the book. Your model of "the book" is rather a fuzzy/approximate state that is updated with new information every time a new word appears. Right? (I'm sorry I know this is very handwavy and psuedoscientific but bear with me). Ok, so if (big if) you feel like the above is true, then to match human-type language modelling, SSMs seem more human-like than transformers. BUT... then aren't transformers strictly better in terms of accuracy? Because a transformer never "forgets" information, as long as it is within the context window, because it revisits that information every time it emits a new token. So let's say we can remove the "quadratic attention" problem of transformers with SSMs. That's a nice training/inference performance boost. But... look at where we got with "naive" attention. GPT 4, Claude 3. It's not like we're hitting a wall with quadratic attention. It's absurdly more expensive than SSMs, but GPUs certainly aren't getting slower. If all AI work stops now, and only hardware improves, it wouldn't be long until GPT4 could run on local hardware, right, provided Moore's law? /end rant, not really sure what my point was, I'm not against SSMs (they're cool) but rather I'm wondering if the SOTA will ever be SSM when attention is so damn good

tippytippytango 2y ago
It's a tradeoff to be managed depending on the application rather than a problem.
maccam912 2y ago
It depends on the task I imagine. Like writing a novel was mentioned, keeping important story lines in your memory for a long time will be necessary, or at least certainly more important than remembering what the characters were eating for lunch on page 10. But if you need to find that one loophole in a contact you probably will benefit from the perfect recall.
spxneo 2y ago
very good point and the sooner we can accept this difference (we access hyperdimensional entities we discover through language and math via fast and slow access and vocalize it through the alphabets we learned to read) the more "intelligence" we can unlock from AI.
aCoreyJ 2y ago
We're running out of the ability to make transistors smaller and closer together so beyond some major breakthrough I wouldnt expect Moore's law to continue nearly long enough to get to the point of running GPT4 on consumer hardware in the short term
- timschmidt 2y ago
  Ah, but we've just begun stacking transistors in the third dimension.
  - jazzyjackson 2y ago
    It was never a solution, Moore's law has more than one dimension as well, not just density but heat dissipation. Can't cool down a transistor that's surrounded by transistors on all sides.
  - ctrw 2y ago

password4321 2y ago

Links to more about Mamba (selective state space models) on HN yesterday: https://news.ycombinator.com/item?id=39853958#39855430 https://news.ycombinator.com/item?id=39853958#39855430

fisian 2y ago
This submission has the same content as the link here (submitted to HN about a month ago): https://news.ycombinator.com/item?id=39501982 https://news.ycombinator.com/item?id=39501982 https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html
- jimmySixDOF 2y ago
  Yes and its the same author this time published on the Gradient (the link before was to the personal blog). The Gradient by the way are amazing curators of AI news in general and have one of the better podcasts I am aware of interviewing developers in the trenches. Adding: this resurgence in Mamba in general is also due to some actual sota progress with SSM like the new AI21 lab released this week [1] and likely to see others merging different architecture layers (this is a 52B MoE with 12B params active during inference blending both Mamba and transformers) >As the first production-grade model based on Mamba architecture, Jamba achieves an unprecedented 3X throughput and fits 140K context on a single GPU. [1] https://www.ai21.com/jamba https://www.ai21.com/jamba

etbebl 2y ago

Anyone else keep seeing articles about Mamba and thinking it's about Python/Conda? It's annoying when the new cool thing picks the same name as something else you like that deserves attention.

ragebol 2y ago
> attention I see what you did there
tempaccount420 2y ago
Sounds like you need a language model to help you categorize Mamba articles into Python and non-Python articles?

Donaldwide 2y ago

[dead]

sp332 2y ago

So in an effective Mamba query the question goes at the end, after input data? I thought that the question should go at the beginning, so it can decide which information in the data is relevant.

eropple 2y ago
I could be wrong, as I haven't used Mamba, but it seems to remain similar to transformers in that it doesn't "decide" anything and streams tokens to follow the existing ones; attention isn't a thing in the same way, but recency does still have impact. To that end, putting context after the question makes it more likely to follow the context, not the question.

jongjong 2y ago

I find it difficult to understand certain math and science papers/articles due to ambiguous use of language. For example "all previous tokens can be passed to the current token." That seems like a poorly constructed sentence. A token is not a function and it's not an algorithm either... How can you pass tokens to a token? This type of ambiguous language in academic papers makes it hard to read... Maybe the phrase 'every token has an association with every other previously encountered token' would be better? Or every token is used to compute the token vector for each token... I don't know, all I can do is guess the meaning of the word 'passed'. They want us to infer and fill in the gaps with our own assumptions. It assumes that we are primed to think in a certain highly constrained way... For some reason a lot of academia around AI is littered with such imprecise language. They choose to use niche concepts and repurposed wording that their own small community invented rather using words and ideas that are more widely understood but which would convey the same information. Rational people who aren't directly involved in those fields who generally resist jumping to conclusions will struggle to understand what is meant because a lot of those words and ideas have different interpretations in their own fields. I studied machine learning at university and wrote ANNs from scratch and trained them and even I find the language and concepts around LLMs too ambiguous. I'd rather just ask ChatGPT. One thing that bothers me is that the community has moved away from relating concepts to neurons, interconnections, input layers, hidden layers and output layers. Instead, they jump straight into vectors and matrices... Pretending as though there is only one way to map those calculations to neurons and weights. But in fact, this abstraction has many possible interpretations. You could have fully connected layers or partially connected layers... Maybe you need a transformer only in front of the input layer or between every layer... So many possibilities. The entire article means little if considered in isolation outside of the context of current configurations of various popular frameworks and tools.

king_magic 2y ago
that's not what it says in the article. it actually says "information from all previous tokens can be passed to the current token". that statement is meaningfully different from "all previous tokens can be passed to the current token". and both really makes sense if you understand attention mechanisms.

ein0p 2y ago

This is more human like, and people will complain that it doesn’t have photographic memory. That is, it’s not superhuman in that regard. But there are many tasks where superhuman recall is not required. We know this because those tasks are currently performed by humans.