4 ms·
Mixture of Experts is already used by pretty much all modern LLMs to address exactly this phenomenon. Hopefully, future models can be trained to be even more a
by NaiveBayesian 2mo ago
Mixture of Experts is already used by pretty much all modern LLMs to address exactly this phenomenon.
Hopefully, future models can be trained to be even more aware of external knowledge, accessible through web search / RAG / whatever it will be then, and might not need to internalize much knowledge at all.
- bbatha 2mo ago> future models can be trained to be even more aware of external knowledge Then you need longer contexts, which is proving to a much more stubborn problem than general knowledge compression.
- tacitusarc 2mo agoNote that MoE is a sparsification mechanism and doesn’t actually have to do with expertise or what would commonly be considered areas of expertise.
- xyzsparetimexyz 2mo agoI suppose the right approach would be to run queries like the ones you want your model to be able to do, sort weights/experts by usage frequency and then reareange the weight so that all but the most frequently used can stay on disk. Tricky though, it could be that a e.g. database question uses 90% of the network at some point or another Edit: I guess at the moment this is just having an LRU cache of experts