5 ms·What 1T parameter base model have you seen from any of those labs?by zackangelo 10mo agoWhat 1T parameter base model have you seen from any of those labs?riku_iki 10mo agoits moe, each expert tower can be branched from some smaller model.jychang 10mo agoThat's not how MoE works, you need to train the FFN directly or else the FFN gate would have no clue how to activate the expert.