You should check out MoE-Mamba (https://arxiv.org/abs/2401.04081 https://arxiv.org/abs/2401.04081), it's faster and more accurate than Transformer-MoE. Of cours

You should check out MoE-Mamba (https://arxiv.org/abs/2401.04081 https://arxiv.org/abs/2401.04081), it's faster and more accurate than Transformer-MoE. Of course only time will tell if it's better when scaled up further than the paper goes.