5 ms·
This is a great idea, but I'm thinking weights aren't enough. If the recent rumors are true of the latest models like chat GPT-4, for example - they have a real
by fullstackchris 3y ago
This is a great idea, but I'm thinking weights aren't enough. If the recent rumors are true of the latest models like chat GPT-4, for example - they have a realitively new style of overlapping _a cohort_ of models, not just utilizing a giant matrix of weights (as has hereforto been the defacto architecture) - or is this still the case even for chat gpt 4 after training?
I'm no expert in ML, so perhaps I am ignorant as to what information is truly needed to reproduce results. I just cant help but think you'd need some hints at the model architecture itself as well.
- famouswaffles 3y agoThere's nothing new about a Mixture of Experts architecture( and it's still essentially a giant matrix of weights). They just consistently well underperformed their dense counterparts so people were surprised. But this recent paper shows MoE seem to benefit a lot better with Instruct training than their dense counterparts. https://arxiv.org/abs/2305.14705 https://arxiv.org/abs/2305.14705