6 ms·
Why aren’t the big models trained on synthetic datasets now? What’s the bottleneck? And how do you avoid amplifying the weaknesses of LLMs when you train on LLM
by keenmaster 3y ago
Why aren’t the big models trained on synthetic datasets now? What’s the bottleneck? And how do you avoid amplifying the weaknesses of LLMs when you train on LLM output vs. novel material from the comparatively very intelligent members of the human species. Would be interesting to see your take on this.
- emadm 3y agoWe are starting to see that, see phi2 for example There are approaches to get the right type of augmented and generated data to feed these models right, check out our QDAIF paper we worked on for example https://arxiv.org/pdf/2310.13032.pdf https://arxiv.org/pdf/2310.13032.pdf