7 ms·
Jamba: Production-grade Mamba-based AI model
- toddmorey 2y agoReleased with open weights!
- kelseyfrog 2y agoI'm glad we're seeing exploration into scaling post-transformer LLM architectures, but I'm disappointed that it has a context window. That was kind of the selling point of Mamba(and SSM models in general), right linear scaling because state+input=next_state+output?
- refulgentis 2y agoI'm not sure I follow fully, it is also the case for (handwaves) "traditional" LLMs that state + input = next state + output. Its just that output increases, so as output becomes input, eventually state + input / next state + output is greater than the context size. Re: linear scaling, that means the runtime cost is O(n) to context size, rather than traditional transformer O(n^2)
- maccam912 2y agoI think kelseyfrog meant that the state for a mamba model is supposed to "remember" stuff even if it doesn't have the actual tokens to reference any more. It might not be guaranteed to hang on to some information about tokens from a long time ago, but at least in theory it's possible, whereas tokens from before a context window in a tradional llms may as well never have existed.
- kelseyfrog 2y agoYes, you said it better than I did :)
- visarga 2y agoThat is valid for Mamba, this model (Jamba) is a mix of transformer and mamba layers, so it still has a quadratic memory cost, but divided by 8.
- a_wild_dandan 2y agostate = context The difference between SSMs and GPTs here is how that state/context scales. Per usual in engineering, there are big trade offs!
- kelseyfrog 2y agoI'm not following. State is a multi-dimensional vector and context is a list of tokens. State is perturbed by A and Bx(t), while context is appended to by sampling the predicted token distribution.
- spxneo 2y ago256k is huge dude. that is like 1/2 of the average non fiction novel i think at least 200~300 pages of PDF im not complaining here and it also fits in GPU
- htrp 2y agocompute still has cost?
- samus 2y agoIn not sure I understood your question. This model should have much lower computational cost since only one out of eight layers is a traditional transformer layer with masked self-attention. Additionally, half of the Mamba layers are MoEs.
- krasin 2y agoThe license is a proper open-source one: Apache 2.0. Thanks, AI21 Labs.
- deleted 2y ago[deleted]
- popalchemist 2y agoIn addition to the architectural and performance benefits, this is the big deal here, IMO.
- spxneo 2y agoim so used to seeing AGPLv3 apache 2 is a more generous license
- krasin 2y agoAGPLv3 is a fine license too. But most of the models nowadays come with bullshit licenses, like Llama 2 with its "acceptable use policy" enforced by the license: https://ai.meta.com/llama/use-policy/ https://ai.meta.com/llama/use-policy/
- Reubend 2y agoIt's great to see a full production level model using Mamba. But when it comes to long context window benchmarks, I'd love to see performance as well as throughput. I was under the impressions that Mamba has huge increases in throughput at the cost of modest losses in accuracy when using long contexts.
- refulgentis 2y agoI would too -- long context has been such a red herring across providers, Claude 3 is the first I've seen that seems to genuinely have some sort of qualitative leap in noticing things. It is worth noting I'm fairly sure there's no inherent theoratical decrease to accuracy in long contexts, the claimed theoratical change is an _increase_ in long-term accuracy in long contexts.
- Arthur_ODC 2y agoLong Context is great and all, but it sucks that all of these LLM's have really poor output length. If I feed something an entire book and ask for a comprehensive summary then I'm expecting at least a full 3-page summary. I get that they try to force these things to be "concise" to save on compute, but good lord it's so annoying.
- CuriouslyC 2y agoThat's a chat gpt problem, if you hit the API it's not nearly so hard to get good output.
- refulgentis 2y agoI wouldn't say that, my latest big user story for making sure I'm handling huge inputs was "translate Moby dick to zoomer". Cant give any service chunks larger than ~5K tokens, over API, without it failing. (Miserably, like, I'd be fine if it gave a paragraph back. But at least on this "map" task, there's a critical point where there's so much input that the reward function ends up imitating the input more instead of chatting)
- gautamcgoel 2y agoWhy include self-attention layers at all? In other words, why not just alternate SSM and MLP layers?
- NLPaep 2y agoMamba is bad with long context. It doesn't remember phone numbers https://www.harvard.edu/kempner-institute/2024/02/05/repeat-after-me-transformers-are-better-than-state-space-models-at-copying/ https://www.harvard.edu/kempner-institute/2024/02/05/repeat-...
- a_wild_dandan 2y agoGood! DNNs unlock semantics (parsing, transforming, producing). That's the basis of general intelligence, not encyclopedic random string recall. Models shouldn't burn ungodly quantities of compute emulating DDR5 with their working memory. We need machines that think better, not memorize well. We already have plenty of those. Massive context windows, and their needle tests, are misguided. We won't reach human-level AGI by basically inventing a natural language RDBMS. Our resources should primarily target better reasoning systems for our models, reinforcement learning, etc. If we can build a GPT4-level problem solving system that coincidentally also can't remember telephone numbers, I'll consider it major progress.
- 6gvONxR4sf7o 2y agoMemorization usually refers to training data. It's often useful to have something that can utilize instructions losslessly, which is the distinction between these models.
- Rodeoclash 2y agoI can't remember phone numbers either but I can use a device suited to remembering them to look them up
- orra 2y agoHell, it looks like you forgot you already said that (-:
- skybrian 2y ago> Jamba boasts an extensive context window of 256K tokens, equivalent to around 210 pages of text, while fitting up to 140K tokens on a single 80GB GPU. I realize this is a big improvement, but it’s striking how inefficient LLM’s are, that you need 80GB of GPU memory to analyze less than 1 megabyte of data. That’s a lot of bloat! Hopefully there’s a lot of room for algorithmic improvements.
- electric_mayhem 2y agoIt’s literally simulating a neural network. How much of your 5-sense experiential memories and decades of academic book learning are you bringing to understand my reply to your post? How many gigabytes do you think that’s equivalent to?
- _false 2y agoI love both parent post perspectives on this.
- skybrian 2y agoJamba seems to be distributed as 21 5-gigabyte files [1] so I guess that’s another way of looking at it. [1] https://huggingface.co/ai21labs/Jamba-v0.1/tree/main https://huggingface.co/ai21labs/Jamba-v0.1/tree/main
- imtringued 2y agoSo what? I have seen models distributed as 26x 10GB files.
- richardw 2y agoIt’s kinda simulating our brains but not really. When I attempted to dig more into how neurons work I realised that it’s a massive chasm of difference. Very much worth doing if you haven’t (you might know far better then me, this is for people who don’t yet.) In terms of results: Our brains are working with 20w of power and can be trained to compete with LLM’s using a tiny fraction of the world’s data. They also have to keep you breathing and your blood pumping and manage all the dangers of catching a ball near traffic. Or skiing, or poetry, or sunsets. And they remember stuff five minutes later and don’t need a training run that takes months. We have SO many opportunities to improve the AI architecture it’s ridiculous. This is a good thing.
- smusamashah 2y agoThere was a recent thread on explaining Mamba https://news.ycombinator.com/item?id=39501982 https://news.ycombinator.com/item?id=39501982 (https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html https://www.kolaayonrinde.com/blog/2024/02/11/mamba.html) There was another one on the same thing, probably better https://news.ycombinator.com/item?id=39482428 https://news.ycombinator.com/item?id=39482428 (https://jackcook.com/2024/02/23/mamba.html https://jackcook.com/2024/02/23/mamba.html)
- dang 2y agoThanks! Macroexpanded: Mamba Explained: The State Space Model Taking On Transformers - https://news.ycombinator.com/item?id=39501982 https://news.ycombinator.com/item?id=39501982 - Feb 2024 (93 comments) Mamba: The Easy Way - https://news.ycombinator.com/item?id=39482428 https://news.ycombinator.com/item?id=39482428 - Feb 2024 (60 comments) Is Mamba Capable of In-Context Learning? - https://news.ycombinator.com/item?id=39286410 https://news.ycombinator.com/item?id=39286410 - Feb 2024 (1 comment) Vision Mamba: Efficient Visual Representation Learning with Bidirectional SSM - https://news.ycombinator.com/item?id=39214939 https://news.ycombinator.com/item?id=39214939 - Feb 2024 (16 comments) MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts - https://news.ycombinator.com/item?id=38932350 https://news.ycombinator.com/item?id=38932350 - Jan 2024 (39 comments) Implementation of Mamba in one file of PyTorch - https://news.ycombinator.com/item?id=38708730 https://news.ycombinator.com/item?id=38708730 - Dec 2023 (109 comments) Show HN: Fortran inference code for the Mamba state space language model - https://news.ycombinator.com/item?id=38687342 https://news.ycombinator.com/item?id=38687342 - Dec 2023 (1 comment) Guide to the Mamba architecture that claims to be a replacement for Transformers - https://news.ycombinator.com/item?id=38659238 https://news.ycombinator.com/item?id=38659238 - Dec 2023 (2 comments) Mamba outperforms transformers "everywhere we tried" - https://news.ycombinator.com/item?id=38606590 https://news.ycombinator.com/item?id=38606590 - Dec 2023 (25 comments) Mamba: Linear-Time Sequence Modeling with Selective State Spaces - https://news.ycombinator.com/item?id=38522428 https://news.ycombinator.com/item?id=38522428 - Dec 2023 (37 comments) Mamba: New SSM arch with linear-time scaling that outperforms Transformers - https://news.ycombinator.com/item?id=38520992 https://news.ycombinator.com/item?id=38520992 - Dec 2023 (2 comments)
- a_wild_dandan 2y agoTo those curious about the tradeoffs between transformer and state space model layers, I highly recommend Sasha Rush's video on it: https://www.youtube.com/watch?v=dKJEpOtVgXc https://www.youtube.com/watch?v=dKJEpOtVgXc
- az226 2y agoThey use less memory for inference but remember the details less well. For instance if you’re implementing code and want edits, it will forget various functions to be part of the script. Even transformers aren’t perfect at this and SSMs are even worse. For many use cases, that ability isn’t needed as much so the memory savings is a bigger lever.
- haddr 2y agoWill it be possible to run such model family in ollama?
- andy99 2y agoMamba is supported in llama.cpp so should be (edit - apparently it's not strictly the mamba architecture, it's a mix of mamba and transformers, so it looks like it would have to be ported to llama.cpp)
- google234123 2y agoI’m pretty sure computational chemists were combining NNs with Kalman Filters for a while now… I recall the issue it was slow due to the N^2 size of the covariance matrix
- ipsum2 2y ago@dang this is blogspam for the official post: https://www.ai21.com/blog/announcing-jamba https://www.ai21.com/blog/announcing-jamba
- deleted 2y ago[deleted]
- ninjahatori 2y agoOn a side note: working over longer contexts also reminds me of MemGPT(https://github.com/cpacker/MemGPT https://github.com/cpacker/MemGPT) I think a similar concept can be applied to Mamba architecture models too.
- eigenvalue 2y agoHas anyone gotten this to work in linux using 1 or 2 4090s? I get stuck on "Loading checkpoint shards: 71%" and then it bails. But weirdly nvidia-smi shows plenty of VRAM available. My machine has 256gb of RAM so I don't think that's the problem either. Really excited to try this one.
- cs702 2y agoPlease link to the original post: https://www.ai21.com/blog/announcing-jamba https://www.ai21.com/blog/announcing-jamba Jamba looks fabulous. Good performance for its size and much more efficient than the available open alternatives. The key idea: One of out of every eight transformer blocks in Jamba applies dot-product attention with quadratic cost, but the other seven out of eight apply a Mamba layer with linear cost. And the entire model is a mixture of experts(MoE) so only ~12B parameters are used at once for inference. Thank you to the folks at AI21 for making Jamba available!
- swyx 2y agoi havent seen anyone mention this yet so i'll be the first - what is the comparison vs StripedHyena? https://www.together.ai/blog/stripedhyena-7b https://www.together.ai/blog/stripedhyena-7b
- cs702 2y agoMamba came out of the same research group, Hazy Research, led by Chris Ré. This new "Jamba" model incorporating Mamba and dot-product attention layers has ~8x more parameters than the largest open Striped Hyena, and appears to work much better.
- deleted 2y ago[deleted]
- sleepingreset 2y agogod damn
- unraveller 2y agoJamba-v0.1-hybrid-MoE (16x6B?) is like giving a big NOS boost to a mixtral 8x7B tier LLM. If true 256k context, 3x longer, faster & cheaper than anything else, it should mean an end to the One Model To Rule Them All mindset for now. The big boys will have to offer some version of it as separate but close side-kick integration to their hero offering.
- moneycantbuy 2y agowould a 192GB RAM mac studio or even a 7950x with 192GB RAM be practical for running this model for inference and possibly fine tuning? Especially if I don't need very low latency e.g. 1 token per second is fine for inference. i also have two 3090s.
- lhl 2y agollama.cpp probably won't be getting Jamba support anytime soon: https://github.com/ggerganov/llama.cpp/issues/6372#issuecomment-2026511099 https://github.com/ggerganov/llama.cpp/issues/6372#issuecomm... There is an MLX Mamba implementation, but nothing for Jamba either: https://github.com/alxndrTL/mamba.py/tree/main/mlx https://github.com/alxndrTL/mamba.py/tree/main/mlx You could run PyTorch on CPU and w/ a 12B activation pass, it might even run relatively fast (8 tok/s?), but a q4 quant would also easily fit on 2x3090s and should run at >60 tok/s.
- throawayonthe 2y ago[dead]
- zelphirkalt 2y agoIs there a Sparabo too? It is always funny to see old names associated with totally different new things!
- CGamesPlay 2y agoDoes this mean that I can continue a chat without needing to send a full transcript? This feels like it could make inference a lot cheaper for multi-step dialogs.
- zzzzzzzzzz10 2y agoWhere can I download and use it?
- kjkjadksj 2y agoPeople need to pick better names. Mamba is already a popular python package and internet search tools are on their knees already.