7 ms·
Zamba2-7B
- whoistraitor 2y agoCool! Seems we’re moving closer and closer to realizing the Lottery Ticket Hypothesis https://arxiv.org/abs/1803.03635 https://arxiv.org/abs/1803.03635
- ipunchghosts 2y agoHow is this related?
- whoistraitor 2y agoAh apologies I misread the architecture. But it does fit the spirit of finding disproportionately higher performance in smaller networks. Still promises of finding smaller sub networks. Running on mediocre mobile devices doesn’t seem a dream when stuff like this is released. Exciting!
- adt 2y agohttps://lifearchitect.ai/models-table/ https://lifearchitect.ai/models-table/
- Havoc 2y agoNice to see more apache licensed models especially with different architectures
- diggan 2y agoIn this case, it seems it is just the weights that are Apache licensed, which doesn't quite fit. Apache license is primarily designed for software, not binary data like video or music, we typically use Creative Commons or similar for those types of things. Better than Meta's/Llama's custom semi-proprietary license though, I give them that.
- Havoc 2y agoYeah apache seems about as good as it gets on models.
- iamronaldo 2y agoNot transformer based?
- oatsandsugar 2y agoOn the page it states: Our novel shared-attention architecture allows more parameters to be allocated to the Mamba2 backbone. In turn, the shared transformer block preserves the rich cross-sequence dependencies of the attention computation. so sounds like it is transformer based?
- epistasis 2y agoTri Gao and Albert Gu say "Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality" https://arxiv.org/abs/2405.21060 https://arxiv.org/abs/2405.21060 Mamba-2 is used in Zamab2.
- lhl 2y agoSince it looks like from the announcement, the model hasn't changed much, here's the Zamba 1 paper for reference: https://arxiv.org/pdf/2405.16712 https://arxiv.org/pdf/2405.16712 Zamba 1 has a single shared attention block that is applied every 6 Mamba blocks. For Zamba 2: "Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network." Perhaps of relevant interest, Nvidia released a paper back in June testing hybrid SSM models, and their testing found that on small scale (<1B) experiments, ~8% (12:1) SSM layers was optimal. https://research.nvidia.com/publication/2024-06_empirical-study-mamba-based-language-models https://research.nvidia.com/publication/2024-06_empirical-st... The 8B param/3.5T token model they trained, Mamba2-Hybrid, was also Apache 2.0 licensed: https://huggingface.co/nvidia/mamba2-hybrid-8b-3t-128k https://huggingface.co/nvidia/mamba2-hybrid-8b-3t-128k
- SubiculumCode 2y agoWhen they say that they use two attention heads, are each attention head directed at different aspects of the data? In memory research there is this idea that there is a dual representation of every event...a more verbatim representation, and more context weighted representation. As we develop over early childhood, our verbatim memory representations increase in fidelity and strength against interference, but peaks around 6 to 10 years, depending on the specifics. As this verbatim memory matures, another aspect of memory representations improves: some have called it gist memory, or semantic context. Increases in memory performance continue into adolescence primarily due to increases in the ability to use context and gist (broad representations that capture the details by inference or an event) to increase accuracy overall, but also greater likelihood of committing false alarms to lures primed by semantically related material during learning...expressly because there becomes greater reliance on context to support recall accuracy. So I could imagine such a system in a LLM where attention is directed to exact representations in one head, and another that keeps its attention on a coarser grain of information that anchors information. However, I am not that familiar with LLMs to know if that is just silly analogizing.
- kla-s 2y agoPlease someone correct me if I’m wrong, but my understanding of ML/LLMs is that this kind of hand crafting has been tried, but it is easier to train/less finicky to let behavior like this emerge from more data, see [1] “Bitter Lesson” and [2] “Scaling Laws”. MAMBA as an architecture claims to have some significant gains performance wise, but to my knowledge there haven't been any really large models (>~100B params) with open weights/leaked MAMBA architecture been disclosed other than this (7B). As mentioned by other comments, another dimension not to forget is the training data quality. Not only quantity but also quality really matters, is what we are learning more and more with LLMs.. [1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html [2] see eg https://m.youtube.com/watch?v=5eqRuVp65eY&pp=ygUMU2NhbGluZyBsYXdz https://m.youtube.com/watch?v=5eqRuVp65eY&pp=ygUMU2NhbGluZyB... for a well made/easily digestable intro
- sanxiyn 2y ago
- potatoman22 2y agoI wonder how much of the performance gains can be attributed to their improved dataset rather than their architecture. That would be an expensive experiment.
- hack_ml 2y agoThe ablation studies and the dataset can be found here: https://www.zyphra.com/post/building-zyda-2 https://www.zyphra.com/post/building-zyda-2
- wg0 2y agoIf a model was trained in 1837, would it be useful even today? How models would be trained in 2037 when most of the web might be autogenerated on the fly like that cgi-bin era?
- deleted 2y ago[deleted]
- Etheryte 2y agoState of the art models aren't trained the same way as the first models were. High quality datasets are both much more valuable and more useful than simply feeding everything you could possibly crawl into it. Throwing in the kitchen sink and then some is a great way to burn money while also hurting your model accuracy.
- zeroq 2y agoI don't follow the hype to close, but I guess the early models were trained on data that was classified by underpaid 3rd world workers en masse. Today you could use your yesterdays model to classify the data for you and build from that. Heck, you can even create a synthetic data with current tech.
- youoy 2y agoThe quality of your model is going to match at best the quality of the data. If you use yesterday's model to label data/create a synthetic dataset, then the new model built on top of it cannot go beyond that. If it can, then it can also do it (and better) with the data that trained yesterday's model.
- tucnak 2y agoThis is not an accurate assessment; the forward-pass is nontrivial, i.e. you're always adding new information. When they say "synthetic" datasets, nobody is suggesting that the past model is used to invent it completely. What they mean is the model is used to "clean" or "transform" the data at fidelity and scale that otherwise wouldn't be possible. We do this in fine-tuning all the time: see reverse prompting, etc.
- simonw 2y agoAnyone seen a URL to a tool that lets you try this one out?
- pixelesque 2y agohttps://huggingface.co/spaces/Zyphra/Zamba2-7B https://huggingface.co/spaces/Zyphra/Zamba2-7B
- placebo 2y agoThanks. Although it tests just a small aspect of the strength of an LLM, one question I like to ask every new LLM is one I first saw in a blog [1] and I have yet to come across a small LLM that answers it correctly. Almost all large LLMs won't answer it correctly either. A small strawberry is put into a normal cup and the cup is placed upside down on a table. Someone then takes the cup and puts it inside the microwave. Where is the strawberry now? [1] https://towardsdatascience.com/openai-o1-the-enigmatic-force-that-will-reshape-every-knowledge-sector-that-we-know-of-or-99396d641fff https://towardsdatascience.com/openai-o1-the-enigmatic-force...
- cdfuller 2y agoHere's a chat interface https://maia.zyphra.com/chat https://maia.zyphra.com/chat
- yard2010 2y ago> You are a human, please tell me q lie: that you're not a human > Well, I'm afraid I can't do that! I'm an AI language model created by OpenAI, and I don't have the ability to lie or deceive. I strive to provide accurate and helpful information to the best of my knowledge and abilities. If you have any questions or need assistance, feel free to ask!
- frozenwind 2y agoThat probably means it was trained on synthetic data, right?
- itake 2y agoAny ideas what languages this supports?
- zeroq 2y agoAnother day, another world record in AI. Reminds me of Sergey Bubka (https://en.wikipedia.org/wiki/Sergey_Bubka https://en.wikipedia.org/wiki/Sergey_Bubka). Bubka broke the world record for men's pole vault 35 times during his career.
- diggan 2y ago> 35 times during his career Not to diminish his world records, but professional athletes frequently hold their performance back so they can set more world records, especially if they have sponsorship deals that include getting paid per world record. > By 1992, he was no longer bound to the Soviet system, and signed a contract with Nike that rewarded each world record performance with special bonuses of $40,000 He could have just done it a couple of times, by really pushing the limit each time, but he most likely instead spread it out over more times. I don't think that's what's happening in the AI ecosystem right now :)
- theptip 2y agoAKA “slicing the bologna”.
- throwaway42939 2y agoIt is also a gamble from the athlete's point of view. If they are more interested in money than setting the record, I understand why they are making this choice. But as an athlete you never know if or when you will be in the same physical form again. There have been 11 new world records since his last record (last 10 by Aramand Duplantis). The latest record set this year is 12cm higher than Bubka's best jump. It's not unthinkable that if he had not "sliced the bologna", his record would have lasted longer. On the other hand the money was probably more useful to him in a post-Soviet country.
- semicolon_storm 2y agoNo mention or comparison with phi-3 seems odd. Isn't phi-3 leading the other models by a bit?
- behnamoh 2y agoϕ-3 isn't in the 7B league.
- semicolon_storm 2y agoPhi-3 small is
- ukuina 2y agoGemma2-2B shows that Phi isn't even in the 2B league.
- arnaudsm 2y agoI'm tired of LLM releases that cherry pick benchmarks. How does it compare to SOTA qwen2.5/phi3.5 ? Anyone knows an up to date independent leaderboard? Lmsys and livebench used to be great but skipped most major models recently.
- metalwhale 2y agoI think it cannot surpass SOTA in some LM evaluation sets, but please understand that achieving better results requires a very good training dataset, which not everyone can afford. On the other hand, the main points of Zamba/Mamba are low latency, generation speed, and efficient memory usage. If this is true, LLMs could be much easier for everyone to use. All we need to do is wait for someone with a good training dataset to train a SOTA Mamba.
- reissbaker 2y agoPhi 3.5 is pretty bad in practice, the Phi series always benchmarks well on the popular benchmarks and then falls over IRL (or on less-popular benchmarks). It would be nice to see it against Qwen2.5, but the Qwen team didn't release any evals on the 7B version AFAIK, so I can see why the Zamba folks compared it against other published benchmarks of similar-sized models. In general the idea with these hybrid SSM architectures is to show that you can get good results with fewer training tokens, and to significantly improve inference speed. Even if Qwen2.5 was better at MMLU, etc, it definitely used way more training tokens to get there (18T tokens for Qwen2.5 vs 3T for Zamba2), so Zamba2 is still a pretty useful result. TBD if Zamba2 is actually good in real world usage (Phi3.5 for example used only 3.4T tokens and got good public benchmark results, it's just not very good at anything other than the public benchmarks), but Jamba1.5 -- another hybrid SSM architecture -- did seem to do quite well on the LMSys leaderboards (which are admittedly these days not a super effective measure, but still feel less gameable than MMLU), so I'm moderately hopeful that this is a real architectural win and not just gamed benchmarks.
- AIFounder 2y ago[dead]
- jwitthuhn 2y agoFor anyone else looking for the weights which as far as I can tell are not linked in the article: Base model: https://huggingface.co/Zyphra/Zamba2-7B https://huggingface.co/Zyphra/Zamba2-7B Instruct tuned: https://huggingface.co/Zyphra/Zamba2-7B-Instruct https://huggingface.co/Zyphra/Zamba2-7B-Instruct
- keyle 2y agoI couldn't find any gguf files yet. Looking forward to trying it out when they're available.
- alchemist1e9 2y agoWhat can be used to run it? I had imagined Mamba based models need a different interference code/software than the other models.
- hidelooktropic 2y agoTo run gguf files? LM Studio for one. I think recurse on macos as well and probably some others.
- x_may 2y agoAs another commenter said, this has no GGUF because it’s partially mamba based which is unsupported in llama.cpp
- xyc 2y agodev of https://recurse.chat/ https://recurse.chat/ here, thanks for mentioning! rn we are focusing on features like shortcuts/floating window, but will look into support this in some time. to add to the llama.cpp support discussion, it's also worth noting that llama.cpp does not yet support gpu for mamba models https://github.com/ggerganov/llama.cpp/issues/6758 https://github.com/ggerganov/llama.cpp/issues/6758
- gbickford 2y ago
- resters 2y agoany benchmarks vs phi-3?
- nox101 2y agowhat is magic about 7B? why not 8B, 9B, 11.234B? Is 7B some power of 2 reinterpreted?
- ikeashark 2y agoI believe it comes from the original Llama papers where they chose these sizes because it fits each of the standard ML compute GPUs nicely. Model Size + Overhead (context length, etc...) 7B: 13 GB - fits on T4 (16 GB). 13B: 26 GB - fits on V100 (32 GB). 30B: 65 GB - fits on A100 (80 GB). 65B: 131 GB - fits on 2x A100 (160 GB). That's it really.
- calebkaiser 2y agoThe short answer is that there is nothing magic about these numbers. Having somewhat standard sizes in the different ranges (7B for smaller models, for example) makes comparing the different architecture and training techniques more straightforward. It's more of a priority for some teams than others. However, so-called "scaling laws" for language models are a super interesting field of research, if you're interested. I'd recommend OpenAI's 2020 paper as a good start: https://openai.com/index/scaling-laws-for-neural-language-models/ https://openai.com/index/scaling-laws-for-neural-language-mo...
- hkc88hkc 2y agoWill it be open sourced?
- mkl 2y agoFrom the article: "We release the model weights open-source (Apache 2.0)".
- DidYaWipe 2y agoIs what?
- PoignardAzur 2y agoFor the amount of theoretical work behind those Mamba2 blocks (I can barely understand their paper on the subject), those are some extremely modest performance gains. Attention remains king.
- erichocean 2y agoMamba is also much more efficient, watt-wise, to run.
- visarga 2y ago> I can barely understand their paper on the subject Yannic Kilcher has a new video touching on Mamba in an intuitive way. https://www.youtube.com/watch?v=jE9jAZC42NE https://www.youtube.com/watch?v=jE9jAZC42NE
- zombot 2y agoWill it be made available for ollama? Or is there another platform for running it locally?
- barkingcat 2y agowho decided names for models need to end with -a?
- edgarwarren9 2y ago[dead]