5 ms·
Excuse my ignorance. Could one just say, "One expert is all I can handle" and strip the others from the model?
by SubiculumCode 2mo ago
Excuse my ignorance. Could one just say, "One expert is all I can handle" and strip the others from the model?
- jeffybefffy519 2mo agoOr could you parallelise your experts on different hardware?
- p-e-w 2mo agoYou could (e.g. by replacing residual-dependent expert routing with hardcoded logic), but quality will suffer dramatically. It’s far better to use a similar-sized dense model then.
- SubiculumCode 2mo agoAhh, I remember hearing that before, and it makes general sense too.
- Infernal 2mo agoI have a similar question and I’m inferring the answer is no - look at the cache hit rate of 23% for the 128GB M5 Max. I had previously assumed that the 40B active meant that a set of layers was chosen as THE expert for a given prompt and generation was then limited to those layers until complete. But in that case you’d have expected the expert caching to have a super high hit rate once you had enough RAM to hold an entire expert’s worth of layers.
- kristianp 2mo agoI you look at https://arxiv.org/pdf/2401.04088 https://arxiv.org/pdf/2401.04088 table 5 on page 8, you'll see that expert(s) used can change from token to token. The experts aren't divided along predictable lines.
- xyzsparetimexyz 2mo agoFor purely coding tasks, is every single expert required? 50%? 25%?
- FeepingCreature 2mo agoYes, every expert. It's not like different personas, more like very coarse dropout training.
- xyzsparetimexyz 2mo agoRight. So it's not like human experts or even different brain regions