9 ms·
Yeah, it's really just trying to get as much compute as possible as cheaply as possible interconnected in a reasonably fast way with low latency. Slow networkin
by ftufek 3y ago
Yeah, it's really just trying to get as much compute as possible as cheaply as possible interconnected in a reasonably fast way with low latency. Slow networking would be a bottleneck and expensive high end networking would defeat the purpose of staying cheap.
- jacquesm 3y agoVery smart approach. I may copy your setup for some project that I've been working on for years but that stalled waiting for more memory in GPUs.
- buildbot 3y agoYou’d be surprised at how cheap high end networking that outperforms PCIE4 x4 is - 100Gb omni-path nics are running for 20$ on ebay! And those will saturate PCIE3 x16. Though of course with multiple boards/ram/cpu it gets complicated again.
- magicalhippo 3y agoWhich cards? Been looking at nics but couldn't find cheap ones past 25-40Gb
- justinclift 3y agoOmni-path is/was the Intel fork of Infiniband, which from rough memory they bought from QLogic some years ago. * Switch: https://www.ebay.com/itm/273064154224 https://www.ebay.com/itm/273064154224 * Adapters: https://www.ebay.com/itm/296188773061 https://www.ebay.com/itm/296188773061 / https://www.ebay.com/itm/166199451199 https://www.ebay.com/itm/166199451199 * Cables: No idea, sorry. ;) * Description: https://www.youtube.com/watch?v=dOIXtsjJMYE https://www.youtube.com/watch?v=dOIXtsjJMYE Note that I don't know those Ebay sellers at all, they're just some of the cheaper results showing up from searching. There seem to be plenty of other results too. :) --- Possibly useful, though it's Proxmox focused: https://forum.level1techs.com/t/proxmox-with-intel-omni-path-fabric-how-to-cautionary-tale/198762 https://forum.level1techs.com/t/proxmox-with-intel-omni-path...