4 ms·
Is there an actual use for this in a homelab? Isn't 25gbps enough that no SSD will saturate it?
by avidiax 13d ago
Is there an actual use for this in a homelab? Isn't 25gbps enough that no SSD will saturate it?
- MarkSweep 13d agoThe second sentence of the article: > We have a number of these now and have been running our RDMA network backend for the 8x NVIDIA GB10 cluster for several months now, using one. I have a friend who uses something similar for his 4 NVIDIA GDX Spark cluster.
- avidiax 13d agoI see, so only if you are running an AI cluster. And what is it transferring between the nodes? Partial computations?
- ericd 13d agoEssentially, yes. The DGX Sparks, at 128 gigs for ~$4700 are one of the cheapest ways to get enough high-ish speed memory cobbled together to run one of the more capable open weight models to run at home/for a small biz. This switch lets you combine the memory of 2-4 of them.
- bitbckt 13d ago4-8 of them. 2 Sparks can operate without a switch.
- ericd 13d agoWell, this one only has 4 ports, would you daisy chain them somehow to get to 8? Or bridge three of these switches?
- dgl 13d agoThey are QSFP56-DD which can be split using breakout cables. So it's 4 x 400GbE ports, which could in theory be split into as many as 32 x 50GbE ports. (Although whether that then ends up being an economical thing to do is a different matter.)
- ericd 13d agoAhh yeah, I've seen people using those. I was thinking you'd ideally want to connect them full speed, but it looks like they're 200gbps ports anyway, and I guess even if they were 400gpbs, you don't really need that much for cross-GPU traffic for MoE models doing tensor parallel.
- nubinetwork 13d agoOh I'm sorry, just let me casually spend 30k...
- MarkSweep 13d agoI'm not defending the use of money; I think it's questionable. It's still a home lab by my definition: it's computer stuff that costs less than a sports car in your home.
- corvad 13d agoI mean at lot of what a homelab is experimenting. For example I have an absurd amount of communication 56GbE Mellanox switches but never really saturate it.
- bobdvb 13d agoYup, I have a pair of CRS504, so I can learn MLAG and my three node cluster doesn't really tickle the sides of that.
- hgoel 13d agoDGX Sparks have 200Gbps interlinks for clustering. Apparently switches are the only practical way to scale such clusters beyond 2 nodes even though each Spark has 2 such ports.
- FireBeyond 13d agoWell, the problem is that you want each node to be able to directly connect to each other, so you run into a mesh situation. https://www.youtube.com/watch?v=QJqKqxQR36Y https://www.youtube.com/watch?v=QJqKqxQR36Y shows 8 running at 100gbps and clustered.
- hgoel 13d agoI understand that part, what I don't quite get is why we can't divide the layers such that each node only needs to talk to at most 2 nodes. I assume I'm misunderstanding how work is distributed between them, but haven't looked into the details yet.
- wtallis 13d agoI think this is the distinction between pipeline parallelism (first N layers on machine 1, next N layers on machine 2, etc.) vs tensor parallelism where all machines contribute to the same matrix multiplication operation within the same layer. Pipeline parallelism doesn't need exotic high speed networking, but also doesn't deliver the kind of speed increase that tensor parallelism provides.
- wtallis 13d agoPlease be careful with units. There are high-end consumer SSDs that will deliver data at 14+GB/s, which is much higher than 25Gb/s.
- toast0 13d agoI purchased my SSD based on price and I have a AM4 APU so I'm limited to pci-e 3, but a quick check with crystaldiskmark says I'm hitting about 3.6 GB/s on sequential reads in seq1m q8t1 ... That would saturate 25gbps (assuming you have the right access patterns). seq1m q1t1 does 2.3 GB/s on my system, close enough to 25 gbps. By my math, pci-e 5.0 should do 4x the throughput, so if you had 4 pci-e ssds you wanted to serve over the network and you had the pci-e lanes, you could saturate 400 gbps. I'm pretty happy with my little baby network with a handful of 10G and 2.5G ports, and I'm cheap so if I made a big upgrade it'd probably be something used that has more 10g ports plus a 40G I could DAC to the adjacent home server ... but my home servers are barely powerful enough to push 10g anyway. (Something like a DCS-7050TX-48 goes for ~ $100 and would be a big upgrade in capacity I don't use over the S3300-28X I currently run)
- FireBeyond 13d agoYes, AI... this video does a good job walking through: https://www.youtube.com/watch?v=QJqKqxQR36Y https://www.youtube.com/watch?v=QJqKqxQR36Y (tl;dw - guy connects first four DGX and then another 4 "DGX-alikes". Uses 400gbps breakout cables (you can split interfaces, TIL) so he has 2 ports on the first switch, and then gets this one, I believe. You can buy cables that are 1x400 to 4x100gig, etc.