5 ms·
They're comparing Qwen's moe vs dense (smaller difference) against Gemma's moe vs dense (bigger difference). Your proposed alternative misses the point.
by Hugsun 5mo ago
They're comparing Qwen's moe vs dense (smaller difference) against Gemma's moe vs dense (bigger difference). Your proposed alternative misses the point.
- zozbot234 5mo agoGemma's dense is bigger than its moe's total parameters. You could totally expect the moe to do terribly by comparison.