7 ms·

Mikrotik has a switch that can do 6x200g for ~$1300 and <150W. https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_crs812_8ds_2dq_2ddq_rm_crs812_ddq_cl

Mikrotik has a switch that can do 6x200g for ~$1300 and <150W. https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_crs812_8ds_2dq_2ddq_rm_crs812_ddq_cloud_router.html/overview?ap=y&smp=Y&srsltid=AfmBOorjdgkxvg6LYd2W9GeOOQ8NXfmJAzGXk5fAejYsIV24uwzikaFfZ4k https://www.bhphotovideo.com/c/product/1926851-REG/mikrotik_...

angoragoats 9mo ago
Cool! So for marginally less in cost and power usage than the numbers I quoted, you can get 2 more machines than with the RDMA setup. And you’ve still not solved the thing that I called out as the most important drawback.
- nicky_nickell 9mo ago
  how significant is the latency hit?
  - angoragoats 9mo ago
    The OP makes reference to this with a link to a GitHub repo that has some benchmarks. TCP over Thunderbolt compared to RDMA over Thunderbolt has roughly 7-10x higher latency, ~300us vs 30-50us. I would expect TCP over 200GbE to have similar latency to TCP over Thunderbolt. Put another way, see the graphs in the OP where he points out that the old way of clustering performs worse the more machines you add? I’d expect that to happen with 200GbE also. And with a switch, it would likely be even worse, since the hop to the switch adds additional latency that isn’t a factor in the TB5 setup.
    - wmf 9mo ago
      You're ignoring RoCE which would have the same or lower latency than RoTB. And I think macOS already supports RoCE.
      - angoragoats 9mo ago
        MacOS does not support RoCE.
    - Hikikomori 9mo ago
      Switch probably does cut through so it starts forwarding the frame before its even fully received.
wtallis 9mo ago
That switch appears to have 2x 400G ports, 2x 200G ports, 8x 50G ports, and a pair of 10G ports. So unless it allows bonding together the 50G ports (which the switch silicon probably supports at some level), it's not going to get you more than four machines connected at 200+ Gbps.
- angoragoats 9mo ago
  As with most 40+GbE ports, the 400Gbit ports can be split into 2x200Gbit ports with the use of special cables. So you can connect a total of 6 machines at 200Gbit.
  - sgjohnson 9mo ago
    Breakout cables typically split to 4. e.g. QSFP28 (100GbE) splits into 4x SFP28s (25GbE each), because QSFP28 is just 4 lanes of SFP28. Same goes for QSFP112 (400GbE). Splits into SFP112s. It’s OSFP that can be split in half, i.e. into QSFPs.
    - angoragoats 9mo ago
      Here’s an example of the cables I was referring to that can split a single 400Gbit QSFP56-DD port to two 200Gbit ports: https://www.fs.com/products/101806.html https://www.fs.com/products/101806.html But all of this is pretty much irrelevant to my original point.
    - buildbot 9mo ago
      This is incorrect - they can split however the driving chip supports. (Q|O)SFP(28|56|112|+) can all be split to a single differential lane. All (Q|O)SFP(28|56|112|+) does is provide basically direct, high quality links to whatever you chips SERDES interfaces can do. It doesn't even have to be ethernet/IB data - I have a SFP module that has a SATA port lol. There's also splitting at the module level, for example I have a PCIe card that is actually a fully self hosted 6 port 100GB switch with it's own onboard Atom management processor. The card only has 2 MPO fiber connectors - but each has 12 fibers, which each can carry 25Gbps. You need a special fiber breakout cable but you can mix anywhere between 6 100GbE ports and 24 25Gbe ports. https://www.silicom-usa.com/pr/server-adapters/switch-on-nic-server-adapters/pe31625g24dira-server-adapter/ https://www.silicom-usa.com/pr/server-adapters/switch-on-nic...
throwaway2037 9mo ago
Wow, this switch (MikroTik CRS812) is scary good for the price point. A quick Google search fails to find any online vendors with stock. I guess it is very popular! Retail price will be <= 1300 USD. I did some digging to find the switching chip: Marvell 98DX7335 Seems confirmed here: https://cdn.mikrotik.com/web-assets/product_files/CRS812-8DS-2DQ-2DDQ_251103.png https://cdn.mikrotik.com/web-assets/product_files/CRS812-8DS... And here: https://cdn.mikrotik.com/web-assets/product_files/CRS812-8DS-2DQ-2DDQ-RM_251055.pdf https://cdn.mikrotik.com/web-assets/product_files/CRS812-8DS... > Switch chip model 98DX7335 From Marvell's specs: https://www.marvell.com/content/dam/marvell/en/public-collateral/switching/marvell-switching-prestera-98dx73xx-product-brief.pdf https://www.marvell.com/content/dam/marvell/en/public-collat... > Description: 32x50G / 16x100G-R2 / 8x100G-R4 / 8x200G-R4 / 4x400G-R8 > Bandwidth: 1600Gbps Again, those are some wild numbers if I have the correct model. Normally, Mikrotik includes switching bandwidth in their own specs, but not in this case.
- cess11 9mo ago
  They are very popular and make quite good products, but as you noticed it can be tricky to find them in stock. Besides stuff like this switch they've also produced pretty cool little micro-switches you can PoE and run as WLAN hotspots, e.g. to distance your mobile user device from some network you don't really trust, or more or less maliciously bridge a cable network through a wall because your access to the building is limited.

angoragoats 9mo ago

Cool! So for marginally less in cost and power usage than the numbers I quoted, you can get 2 more machines than with the RDMA setup. And you’ve still not solved the thing that I called out as the most important drawback.

nicky_nickell 9mo ago
how significant is the latency hit?
- angoragoats 9mo ago
  The OP makes reference to this with a link to a GitHub repo that has some benchmarks. TCP over Thunderbolt compared to RDMA over Thunderbolt has roughly 7-10x higher latency, ~300us vs 30-50us. I would expect TCP over 200GbE to have similar latency to TCP over Thunderbolt. Put another way, see the graphs in the OP where he points out that the old way of clustering performs worse the more machines you add? I’d expect that to happen with 200GbE also. And with a switch, it would likely be even worse, since the hop to the switch adds additional latency that isn’t a factor in the TB5 setup.
  - wmf 9mo ago
    You're ignoring RoCE which would have the same or lower latency than RoTB. And I think macOS already supports RoCE.
    - angoragoats 9mo ago
      MacOS does not support RoCE.
  - Hikikomori 9mo ago
    Switch probably does cut through so it starts forwarding the frame before its even fully received.

wtallis 9mo ago

That switch appears to have 2x 400G ports, 2x 200G ports, 8x 50G ports, and a pair of 10G ports. So unless it allows bonding together the 50G ports (which the switch silicon probably supports at some level), it's not going to get you more than four machines connected at 200+ Gbps.

angoragoats 9mo ago
As with most 40+GbE ports, the 400Gbit ports can be split into 2x200Gbit ports with the use of special cables. So you can connect a total of 6 machines at 200Gbit.
- sgjohnson 9mo ago
  Breakout cables typically split to 4. e.g. QSFP28 (100GbE) splits into 4x SFP28s (25GbE each), because QSFP28 is just 4 lanes of SFP28. Same goes for QSFP112 (400GbE). Splits into SFP112s. It’s OSFP that can be split in half, i.e. into QSFPs.
  - angoragoats 9mo ago
    Here’s an example of the cables I was referring to that can split a single 400Gbit QSFP56-DD port to two 200Gbit ports: https://www.fs.com/products/101806.html https://www.fs.com/products/101806.html But all of this is pretty much irrelevant to my original point.
  - buildbot 9mo ago
    This is incorrect - they can split however the driving chip supports. (Q|O)SFP(28|56|112|+) can all be split to a single differential lane. All (Q|O)SFP(28|56|112|+) does is provide basically direct, high quality links to whatever you chips SERDES interfaces can do. It doesn't even have to be ethernet/IB data - I have a SFP module that has a SATA port lol. There's also splitting at the module level, for example I have a PCIe card that is actually a fully self hosted 6 port 100GB switch with it's own onboard Atom management processor. The card only has 2 MPO fiber connectors - but each has 12 fibers, which each can carry 25Gbps. You need a special fiber breakout cable but you can mix anywhere between 6 100GbE ports and 24 25Gbe ports. https://www.silicom-usa.com/pr/server-adapters/switch-on-nic-server-adapters/pe31625g24dira-server-adapter/ https://www.silicom-usa.com/pr/server-adapters/switch-on-nic...

throwaway2037 9mo ago

Wow, this switch (MikroTik CRS812) is scary good for the price point. A quick Google search fails to find any online vendors with stock. I guess it is very popular! Retail price will be <= 1300 USD. I did some digging to find the switching chip: Marvell 98DX7335 Seems confirmed here: https://cdn.mikrotik.com/web-assets/product_files/CRS812-8DS-2DQ-2DDQ_251103.png https://cdn.mikrotik.com/web-assets/product_files/CRS812-8DS... And here: https://cdn.mikrotik.com/web-assets/product_files/CRS812-8DS-2DQ-2DDQ-RM_251055.pdf https://cdn.mikrotik.com/web-assets/product_files/CRS812-8DS... > Switch chip model 98DX7335 From Marvell's specs: https://www.marvell.com/content/dam/marvell/en/public-collateral/switching/marvell-switching-prestera-98dx73xx-product-brief.pdf https://www.marvell.com/content/dam/marvell/en/public-collat... > Description: 32x50G / 16x100G-R2 / 8x100G-R4 / 8x200G-R4 / 4x400G-R8 > Bandwidth: 1600Gbps Again, those are some wild numbers if I have the correct model. Normally, Mikrotik includes switching bandwidth in their own specs, but not in this case.

cess11 9mo ago
They are very popular and make quite good products, but as you noticed it can be tricky to find them in stock. Besides stuff like this switch they've also produced pretty cool little micro-switches you can PoE and run as WLAN hotspots, e.g. to distance your mobile user device from some network you don't really trust, or more or less maliciously bridge a cable network through a wall because your access to the building is limited.