6 ms·
Zebra-Llama – Towards efficient hybrid models
- deleted 10mo ago[deleted]
- mason_mpls 10mo ago> Zebra-Llama achieves Transformer-level accuracy with near-SSM efficiency using only 7–11B training tokens (compared to trillions of tokens required for pre-training) and an 8B teacher. Moreover, Zebra-Llama dramatically reduces KV cache size—down to 3.9%, 2%, and 2.73% of the original for the 1B, 3B, and 8B variants, respectively—while preserving 100%, 100%, and 97% of average zero-shot performance on LM Harness tasks. This is an extraordinary claim, is there a catch I’m missing? Am I misreading?
- jychang 10mo agoThe catch that you're missing is that Deepseek did this ages ago. They're just using MLA, which is well known to reduce KV size by 90%. You know, the MLA that's used in... Deepseek V2, Deepseek V3, Deepseek R1, Deepseek V3.1, Deepseek V3.2. Oh, and they also added some hybrid linear attention stuff to make it faster at long context. You know who else uses hybrid linear attention? Deepseek V3.2.
- erichocean 10mo agoKimi K2 also uses MLA, and Kimi Linear runs Kimi Delta Attention (it's SSM-like) for three out of every four layers (the fourth uses MLA).
- jychang 10mo agoKimi K2 is literally a "copy Deepseek's homework" model. Seriously. It's even exactly 61 layers, the same as Deepseek V3/R1.
- logicprog 10mo agoFor a "copy Deepseek's homework" model, it's really good, preferable to DeepSeek for me (at least prior to V3.2, which I haven't been able to fully put through its paces yet). post-training really makes that much of a difference I guess
- storus 10mo agoLinear attention is really bad, it's only good for benchmaxing but it leads to a loss of valuable granularity, which can be felt in the latest DeepSeek randomly forgetting/ignoring/correcting explicitly stated facts in the prompt.
- adityashankar 10mo agoDue to perverse incentives and the historical nature of models over-claiming accuracy, it's very hard to believe anything until it is open source and can be tested out that being said, I do very much believe that computational efficiency of models is going to go up [correction] drastically over the coming months, which does pose interesting questions over nvidia's throne *previously miswrote and said computational efficiency will go down
- danielbln 10mo agoI think you mean computational efficiency will go _up_ in the future. To your last point: Jevons paradox might apply.
- adityashankar 10mo agoyup that's what I meant!, Jevon's paradox applies to resource usage in general and not towards a specific companies dominance if computational efficiency goes up (thanks for the correction), and CPU inference becomes viable for most practical applications, GPUs (or accelerators) themselves may be unnecessary for most practical functions
- atq2119 10mo agoDiscrete GPUs still have an advantage in memory bandwidth. Though this might push platforms like laptops towards higher bandwidths, which would be nice.
- credit_guy 10mo agoLike this? https://huggingface.co/amd/Zebra-Llama-8B-8MLA-24Mamba-SFT https://huggingface.co/amd/Zebra-Llama-8B-8MLA-24Mamba-SFT
- deepdarkforest 10mo ago> which does pose interesting questions over nvidia's throne... > Zebra-Llama is a family of hybrid large language models (LLMs) proposed by AMD that... Hmmm
- Reubend 10mo agoIt would be REALLY cool to see this same technique applied to a much more recent OSS model distillation. For example, Mistral 3 14B would be a great target. How efficient can we get inference there?
- a_wild_dandan 10mo agoIf the claims in the abstract are true, then this is legitimately revolutionary. I don’t believe it. There are probably some major constraints/caveats that keep these results from generalizing. I’ll read through the paper carefully this time instead of a skim and come back with thoughts after I’ve digested it.
- jychang 10mo agoWhat's not to believe? Qwerky-32b has already done something similar as a finetune of QwQ-32b but not using traditional attention architecture. And hybrid models aren't new, MLA based hybrid models is basically just Deepseek V3.2 in a nutshell. Note that Deepseek V3.2 (and V3.1, R1, and V3... and V2 actually) all use MLA. Deepseek V3.2 is what adds the linear attention stuff. Actually, since Deepseek V3.1 and Deepseek V3.2 are just post-training on top of the original Deepseek V3 pretrain run, I'd say this paper is basically doing exactly what Deepseek V3.2 did in terms of efficiency.
- cubefox 9mo agoDeepSeek-V3.2 is a sparse attention architecture, while Zebra-Llama is a hybrid attention/SSM architecture. The outcome might be similar in some ways (close to linear complexity) but I think they are otherwise quite different.
- xer 10mo agoThis is great! But what if the US invests 1% of GDP in GPU datacenters and then those are not needed becaues someone created a much more efficient architecture?
- _boffin_ 10mo agoThey will still use capacity. Why would you believe anything different?
- wild_egg 10mo agoMore efficiency just means more consumption. Think when they add lanes to a highway, traffic gets better for a little bit but very soon the highway is just as congested as before.
- wilg 10mo agoMore people get where they’re going in the same amount of time though
- dkural 10mo agoLook up Jevons Paradox, when something becomes more efficient, consumption can goes up, often due to price elasticity. Think of like this: Imagine car prices go from $200,000 to $$20,000 - you wouldn't sell 10x the amount of cars, you'd sell --- In fact I just looked up the numbers - worldwide only 100K or so cars are 200K & higher, whereas roughly 80 million cars are in that affordable category. So a price drop of 90% allowed sales to go from 0.1M to 80M!! I think this means we need more engines, tires, roads, gas, spare parts.
- chpatrick 10mo agoThen they'll be able to use those datacenters much more efficiently.
- AlexCoventry 10mo agoThis is from May 2025, according to the arxiv watermark. Maybe that should be mentioned in the title.
- KnuthIsGod 10mo agoLooks like the trillions of dollars spent on datacentres will end up being regretted.
- pryelluw 10mo agoI should have been an electrician.