5 ms·
Will it be possible to do LLM Pool Training?
Inspired by Crypto Pool Mining: would it be possible to train LLMs as a pool of volunteers? I feel like that would be a good way to have frontier models that are open and don’t rely on big corporations with data centers to train and release them…
- verdverm 2mo agopossible, yes, people already have projects practical, not yet, if ever (?) the difference is data and calculation dependencies, in crypto they are independent
- bjourne 2mo agoDatacenter-scale GPGPU compute is extremely efficient compared to what you would achieve with a network of independently operated distributed heterogeneous devices spread across the world. Just imagine the overhead for ensuring that a node doesn't poison the gradients. And that's just one very small part of the problem.