7 ms·
Well, the problem is that you want each node to be able to directly connect to each other, so you run into a mesh situation. https://www.youtube.com/watch?v=QJq
by FireBeyond 13d ago
Well, the problem is that you want each node to be able to directly connect to each other, so you run into a mesh situation. https://www.youtube.com/watch?v=QJqKqxQR36Y https://www.youtube.com/watch?v=QJqKqxQR36Y shows 8 running at 100gbps and clustered.
- hgoel 13d agoI understand that part, what I don't quite get is why we can't divide the layers such that each node only needs to talk to at most 2 nodes. I assume I'm misunderstanding how work is distributed between them, but haven't looked into the details yet.
- wtallis 13d agoI think this is the distinction between pipeline parallelism (first N layers on machine 1, next N layers on machine 2, etc.) vs tensor parallelism where all machines contribute to the same matrix multiplication operation within the same layer. Pipeline parallelism doesn't need exotic high speed networking, but also doesn't deliver the kind of speed increase that tensor parallelism provides.