5 ms·
I am sorry I did not understand all of it. But, would this allow running large MoE LLMs on a local network with experts spread out over multiple cheaper GPUs (o
by brainless 1mo ago
I am sorry I did not understand all of it. But, would this allow running large MoE LLMs on a local network with experts spread out over multiple cheaper GPUs (or even CPUs)? This would perhaps be more useful than over the Internet, within offices for example.
- vforno 1mo agoThat’s one of the strongest use-cases. On a local network (office, lab, home cluster) the RTT is a few milliseconds instead of 20-50 ms, so the expert-offloading becomes much more practical. You can spread the experts across several cheaper GPUs or even CPUs, keep only the dense parts + router on the machine you’re chatting from, and the whole thing stays private inside your LAN. No internet required, no cloud, just the machines you already have.