6 ms·
And with MTP (or other speculation techniques) you can ~double that.
by SlavikCA 3mo ago
And with MTP (or other speculation techniques) you can ~double that.
- phamilton 3mo agoMTP on a MoE is hit or miss. If you're bottlenecked on memory, MTP can increase the number of active experts (like any batch processing would), which can eat away gains from it.