6 ms·
Since it's a MoE model with 37B active params, I imagined you don't even need all of that ram to keep the whole model in memory, just the active bits.
by guntars 2y ago
Since it's a MoE model with 37B active params, I imagined you don't even need all of that ram to keep the whole model in memory, just the active bits.
- rahimnathwani 2y agoThe active bits may change with each token. You need the whole model in memory, even though, for any single token, only a subset of that memory will have been used in computation. The memory efficiency comes when you have multiple sessions in parallel.