19 ms·

I meant each card is limited to 240w, instead of the usual 450w. Also, it's more like 4 circuits after all, because the main cpu/mb/2gpus are on a 15a too.

by ftufek 3y ago

jacquesm 3y ago
Ah! Ok, thank you now I get it. That's a very nice rig you have there. So at a guess you didn't care as much about the peak computing capacity as long as whatever you are doing all fits in GPU memory and this is your way of collecting that much memory in a single machine so you still have reasonable interconnect speeds between GPUs?
- ftufek 3y ago
  Yeah, it's really just trying to get as much compute as possible as cheaply as possible interconnected in a reasonably fast way with low latency. Slow networking would be a bottleneck and expensive high end networking would defeat the purpose of staying cheap.
  - jacquesm 3y ago
    Very smart approach. I may copy your setup for some project that I've been working on for years but that stalled waiting for more memory in GPUs.
  - buildbot 3y ago
    You’d be surprised at how cheap high end networking that outperforms PCIE4 x4 is - 100Gb omni-path nics are running for 20$ on ebay! And those will saturate PCIE3 x16. Though of course with multiple boards/ram/cpu it gets complicated again.
    - magicalhippo 3y ago
      Which cards? Been looking at nics but couldn't find cheap ones past 25-40Gb
      - justinclift 3y ago
        Omni-path is/was the Intel fork of Infiniband, which from rough memory they bought from QLogic some years ago. * Switch: https://www.ebay.com/itm/273064154224 https://www.ebay.com/itm/273064154224 * Adapters: https://www.ebay.com/itm/296188773061 https://www.ebay.com/itm/296188773061 / https://www.ebay.com/itm/166199451199 https://www.ebay.com/itm/166199451199 * Cables: No idea, sorry. ;) * Description: https://www.youtube.com/watch?v=dOIXtsjJMYE https://www.youtube.com/watch?v=dOIXtsjJMYE Note that I don't know those Ebay sellers at all, they're just some of the cheaper results showing up from searching. There seem to be plenty of other results too. :) --- Possibly useful, though it's Proxmox focused: https://forum.level1techs.com/t/proxmox-with-intel-omni-path-fabric-how-to-cautionary-tale/198762 https://forum.level1techs.com/t/proxmox-with-intel-omni-path...

I meant each card is limited to 240w, instead of the usual 450w. Also, it's more like 4 circuits after all, because the main cpu/mb/2gpus are on a 15a too.

by ftufek 3y ago

I meant each card is limited to 240w, instead of the usual 450w. Also, it's more like 4 circuits after all, because the main cpu/mb/2gpus are on a 15a too.

jacquesm 3y ago

Ah! Ok, thank you now I get it. That's a very nice rig you have there. So at a guess you didn't care as much about the peak computing capacity as long as whatever you are doing all fits in GPU memory and this is your way of collecting that much memory in a single machine so you still have reasonable interconnect speeds between GPUs?

ftufek 3y ago
Yeah, it's really just trying to get as much compute as possible as cheaply as possible interconnected in a reasonably fast way with low latency. Slow networking would be a bottleneck and expensive high end networking would defeat the purpose of staying cheap.
- jacquesm 3y ago
  Very smart approach. I may copy your setup for some project that I've been working on for years but that stalled waiting for more memory in GPUs.
- buildbot 3y ago
  You’d be surprised at how cheap high end networking that outperforms PCIE4 x4 is - 100Gb omni-path nics are running for 20$ on ebay! And those will saturate PCIE3 x16. Though of course with multiple boards/ram/cpu it gets complicated again.
  - magicalhippo 3y ago
    Which cards? Been looking at nics but couldn't find cheap ones past 25-40Gb
    - justinclift 3y ago
      Omni-path is/was the Intel fork of Infiniband, which from rough memory they bought from QLogic some years ago. * Switch: https://www.ebay.com/itm/273064154224 https://www.ebay.com/itm/273064154224 * Adapters: https://www.ebay.com/itm/296188773061 https://www.ebay.com/itm/296188773061 / https://www.ebay.com/itm/166199451199 https://www.ebay.com/itm/166199451199 * Cables: No idea, sorry. ;) * Description: https://www.youtube.com/watch?v=dOIXtsjJMYE https://www.youtube.com/watch?v=dOIXtsjJMYE Note that I don't know those Ebay sellers at all, they're just some of the cheaper results showing up from searching. There seem to be plenty of other results too. :) --- Possibly useful, though it's Proxmox focused: https://forum.level1techs.com/t/proxmox-with-intel-omni-path-fabric-how-to-cautionary-tale/198762 https://forum.level1techs.com/t/proxmox-with-intel-omni-path...