12 ms·
Because something linear like Mamba doesn't perform as well; so you'd have a performance cliff, where suddenly the model would get more dumb and forget a lot of
by lambda 6mo ago
Because something linear like Mamba doesn't perform as well; so you'd have a performance cliff, where suddenly the model would get more dumb and forget a lot of what was going on.
Instead, you can get benefits from both by doing both in parallel. This can let you reduce the size of the O(n^2) attention mechanism, so while it's still quadratic, it reduces the constant quite a bit while still retaining a lot of performance, as the linear context mechanism can work for the tasks its well suited for while allowing attention to play to its strengths.
The recent Nemotron 3 Nano and Super models from NVIDIA are hybrid architectures this way, with most of their context layers as Mamba while retaining enough attention to continue to be competitive on the more complex tasks that require the quadratic attention.
See https://magazine.sebastianraschka.com/i/168650848/18-nemotron-3-nano-and-super https://magazine.sebastianraschka.com/i/168650848/18-nemotro... for some discussion on this architecture
- 3abiton 6mo agoI am curious of the tradeoff of hybrid approaches, it sounds too good to be true.
- lambda 6mo agoIt mostly trades some potential performance loss for speed, especially at longer contexts. Nemotron 3 Super doesn't perform quite as well on benchmarks as the similarly sized Qwen3.5 122B A10B model, but it goes faster and is cheaper to run. https://artificialanalysis.ai/?models=gpt-oss-120b%2Cmistral-small-4%2Cnvidia-nemotron-3-super-120b-a12b%2Cqwen3-5-122b-a10b https://artificialanalysis.ai/?models=gpt-oss-120b%2Cmistral... Now, you're not exactly comparing apples to apples there, since the training process (mix of data for pre-training, and the fine tuning stages of instruction turning, RLVR, etc) could have as much or more impact on how well it does as the architecture itself. Nemotron 3 Super does get better scores on performance than GPT-OSS 120B and Mistral Small 4, both also similarly sized open weights models.