7 ms·
What are you talking about? Why would you multiply the bandwidth? 8 4090s is still 1000GB/s. While the M2 Ultra is 800GB/s with a top of 192GB VRAM. Metal can a
by klohto 2y ago
What are you talking about? Why would you multiply the bandwidth? 8 4090s is still 1000GB/s. While the M2 Ultra is 800GB/s with a top of 192GB VRAM. Metal can access ~155GB, so you need a bit more, but your comparison makes absolutely zero sense.
- boroboro4 2y agoThere are different ways to run LLMs on multiple GPUs, one of them (called tensor parallelism) in low batch scenarios would be multiplying bandwidth between different GPUs. So no, 8 4090s is not 1000 GB/s.
- klohto 2y agolet me know how is the PCIe bandwidth treating you
- boroboro4 2y agoSince we’re talking about small batch sizes PCIe bandwidth isn’t as important - intermediate hidden state is magnitude smaller than weights.
- behnamoh 2y agoyou've heard something and are regurgitating it without fully understanding it.
- boroboro4 2y agoI’m developing inference engine, so I actually do understand how it works. As well as other types of parallelism and how exactly they do different trade offs
- andersa 2y agoWhen you have 8 GPUs, you can use more than 1 at a time.