10 ms·
USB4/thunderbolt is a magical protocol. Turns out the fastest way to move data between 2 modern PCs is to connect their thunderbolt ports with a USB-C cable. Th
by gaudat 3y ago
USB4/thunderbolt is a magical protocol. Turns out the fastest way to move data between 2 modern PCs is to connect their thunderbolt ports with a USB-C cable. The connection shows up as a ethernet port on Windows and I can easily saturate a SSD with its 1GB/s+ transfer rate. And it just works (tm). Reminds me of firewire on these 20 yo Macs.
And What happen if you wire both ports together on the same PC?.. Do you get a broadcast (thunder) storm?
- ComputerGuru 3y ago> The connection shows up as a ethernet port on Windows Do you know if this is the case for all thunderbolt generations (speed differences aside)? Does it apply to thunderbolt using mini DisplayPort too or only over USB PHY?
- vineyardmike 3y agoI think it works for older thunderbolt too. Been years since I tested it though.
- chx 3y agoIt was possible on Thunderbolt 2 , on the Mac side OS X Mavericks enabled it, this Intel whitepaper from 2014 talks about the Windows side. https://www.thunderbolttechnology.net/sites/default/files/Thunderbolt%E2%84%A2%20Networking%20Bridging%20and%20Routing%20Instructional%20White%20Paper.pdf https://www.thunderbolttechnology.net/sites/default/files/Th... https://www.gigabyte.com/Press/News/1140 https://www.gigabyte.com/Press/News/1140 this doesn't mention it and the Intel whitepaper specifically requires TB2 so I would guess TB2 was it.
- raffraffraff 3y agoPity Thunderbolt 2 is basically non-existent nowadays. I have a few Macbook Pro 13 (2015) and I'd love to be able to use the thunderbolt 2 ports, but peripherals were too expensive and the standard short-lived. Try finding a thunderbolt 2 dock anywhere. Filter out all the false-positives (USB-C docks) and the are maybe 10 total on ebay, and they're stupidly expensive for 6+ year old used devices, most without cables or power supplies. Such a pity because they can really extend the useful life of those laptops.
- russelg 3y agoTB3 is backwards compatible so you can use the Apple TB2 to 3 adapter in conjunction with a TB2 cable to hook up any TB3 device to your MBP 2015. I had the mid-2015 15" MBP and used the adapter to hook up an external GPU. If that can work I'm sure a TB3 dock will.
- ComputerGuru 3y agoIt’s the beauty and elegance of the PCIe design. Thunderbolt just provides convenient ported access to those lanes.
- chx 3y agoThe cardinal sin of USB IF was releasing the 5gbps then 10gbps USB modes. It should've been PCIe 2.0 x1 and then 3.0 x1. There was absolutely no reason not to do it: PCIe 2.0 came out in January 2007, USB 3.0 came out in November 2008. PCIe 3.0 followed in November 2010 and 10gbps over the USB C connector didn't appear until August 2014. What USB4 version 2.0 can only do with a complex tunneling architecture we could get "straight": PCIe 5.0 x1 can do 32gbps which closely matches the 40gbps lane speed defined in USB4 version 2.0 (which again came out years after PCIe 5.0 mind you). It would require two lanes, one for RX one for TX and the other two lanes could carry UHBR20 data for display, for a total of 40gbps. This very closely resembles the 80gbps bus speed of USB4 version 2.0 but the architecture is vastly simpler. We wouldn't have needed dubious quality separate USB-to-SATA then USB-to-Ethernet etc adapters. External 10GbE would be ubiqutious instead of barely existing and expensive. Similarly, eGPUs would not need to be a niche and DisplayLink simply wouldn't exist because it wouldn't need to exist and the world would be a better place for it. You could just run a very low wattage very simple but real GPU instead. Say, the SM750 is like 2W.
- ComputerGuru 3y agoI get what you’re saying but not specifically how you’re imagining the implementation. What do you envision the difference between thunderbolt and usb would be in this case? All complex/bandwidth-intensive applications would be better suited to use PCIe directly, but the problem has always been that for various peripherals this imposes a (small) cost manufacturers would rather not pay and would prefer to have the usb spec abstract over.
- davkan 3y agoMy TB3 mesh network shows interfaces as thunderbolt0 etc. this is on Linux using thunderbolt _net from the kernel. Latency is worse than regular twisted pair Ethernet.
- bobim 3y agoDang, that wouldn’t play nice for MPI then.
- davkan 3y agoI was seeing 1-1.5ms latency using linux bridges for the mesh. Not a huge issue for ceph replication but significantly more than switched lan. It may be possible to get it lower with routed instead of bridged but my understanding is thunderbolt_net on Linux is not perfect in that regard.
- jwells89 3y agoTarget disk mode over FireWire was magical back in the day. Nothing like turning someone’s laptop into an oversized external hard drive to rescue data or get a borked OS X installation booting again.
- xattt 3y agoFireWire had IP networking stacks in Windows and OS X in 2007. You could daisy chain a bunch of devices together to share a network connection.
- FirmwareBurner 3y agoSure but it was very expensive compared to USB and Ethernet so Firewire never caught on with mainstream conumers other than some niche cases like camcorders. Thunderbolt was also expensive, which is why adoption was limited, but it's becoming more maisntream since Intel and Apple have been pushing it in the last years, adn piggibacking over USB-C makes it an easy sell comapred to requireing a separate connector like firewire. Still, thunderbolt peripherals are way more expensive than USB ones, so like Firewire before, use is still more in the enthusianst/professional space.
- GeekyBear 3y ago> it was very expensive compared to USB and Ethernet so Firewire never caught on Compared to Gigabit Ethernet back in that time period? Firewire was a huge bargain.
- deleted 3y ago[deleted]
- FirmwareBurner 3y agoNo, not gabit, but 100 Ethernet was more than enough for what average consumers had to transfer back then, and it was significantly cheaper and more available than firewire. It was more likely your HDD to be a bottleneck for faster network transfers.
- vardump 3y agoYou need more like 50 Gbps to saturate a modern Nvme SSD. 10 Gbps doesn't come close.
- genman 3y agoThere was a moment with the spinning rust drives where it would have made sense to have storage in a networked device and not locally but now it rarely makes sense unless an incredibly fast interconnect could be used. Of course this example is still interesting and cool.
- arglebargle123 3y agoFor a couple of years I had a Linux NAS box under my desk with like 8 Samsung 850 pros in a big array connected to my desktop over 40GbE. Then NVMe became a common thing and the complexity wasn't worthwhile.
- NavinF 3y agoInfiniband can match NVMe bandwidth and its latency is similar. Newer network cards can also present a NVMe-oF drive as a local NVMe drive
- mgerdts 3y ago25 Gb Ethernet roughly matches a PCIe Gen 3 NMVe drive’s max throughput 50 Gb will match Gen 4. These are RDMA capable. It seems 25Gb dual port Mellanox CX-4 cards can be found on eBay for about $50. The cables will be a bit pricey. If not doing back to back, the switch will probably be very pricey.
- genman 3y agoIt can match a single drive.
- mgerdts 3y agoAssuming jumbo packets are used with RoCE, every 4096 bytes of data will have 70 bytes of protocol overhead [1]. This means that a 25 Gb/s Ethernet link can deliver no more than 3.07 GB/s of throughput. Each lane of PCIe Gen 3 can deliver 985 MB/s [2], meaning the typical drive that uses 4 lanes would max out at 3.9 GB/s. Surely there is some PCIe/NVMe overhead, but 3.5 GB/s is achievable if the drive is fast enough. There are many examples of Gen 4 drives that deliver over 7 GB/s. Supposing NVMe-oF is used, the MVMe protocol overhead over Ethernet and PCIe will be similar. 1. https://enterprise-support.nvidia.com/s/article/roce-v2-considerations https://enterprise-support.nvidia.com/s/article/roce-v2-cons... 2. https://en.wikipedia.org/wiki/PCI_Express https://en.wikipedia.org/wiki/PCI_Express
- znpy 3y ago> And What happen if you wire both ports together on the same PC? A literal loopback interface? Two of them, most likely.
- samstave 3y agoWTF! I have two identical machines that I need to do this with... lemme test it and see. ... ARGs usb-3... so nopes. (How FN lame is that)
- drewzero1 3y agoIndeed... I just got my first computer with a USB-C port and have been puzzling over what to do with it. Most of the cool tricks seem to require Thunderbolt, which it is not.
- SloopJon 3y agoI've been playing with Thunderbolt networking over the past week with mixed results. I can get 16Gbps between a couple of Macs. Between a Mac and a PC running Windows 10 I get similar speeds in one direction, but less than 1Gbps in the other direction. In terms of scaling this to multiple hosts as the author does, I've read that it is possible to daisy chain, or even use a hub, but it doesn't strike me as the most reliable way to build a network. For an ad hoc connection, though (like null modem cables of yore), it's a great option.
- belthesar 3y agoI think reliability is a great metric to evaluate this on. I don't have a lot of experience with USB4/Thunderbolt networking, but as far as ring network principals go, when you have a network with only 3 nodes, a ring topology is also a fully connected topology. This means that connectivity between nodes should never fail due to the failure of a node. That screams reliable to me. As far as points of failure, there's no additional hub/switch in between the devices, so you have a Thunderbolt controller on each device, two cables, and two ports. If a cable goes bad, so long as there isn't a silent/awkward failure mode, all three nodes can still talk to eachother, at degraded speed. If a switch goes bad, the whole network is down, unless you start talking about redundant switch topologies. To your point though, there does seem to be plenty of shenanigans with performance, especially between devices with different Thunderbolt controllers, that may make this less ideal. But IMO, that's more of a question of do you want to with a more battle tested topology, or are you okay with a less battle tested, but still highly performant and "simple" (we won't go into how bonkers the USB/Thunderbolt spec is) topology?
- phyzome 3y ago> The connection shows up as a ethernet port OK but what then? I've had ethernet ports on my computers since I can remember, and that hasn't magically allowed me to transfer data back and forth just by plugging a patch cable into both machines. What software is at work here?
- bpye 3y agoI’m guessing if they’re using Windows it’s boring old SMB?
- SloopJon 3y agoDepending on what you want to do on what O/S, batteries may or may not be included. I use Moonlight on a Mac to control a Windows 10 PC running Sunshine--lower latency than Remote Desktop (which I don't have anyway in the Home edition), and nicer looking than Parsec. When you connect a patch cable directly (no crossover cable needed in the 21st century), you'll likely find that each system has a self-assigned IP in the 169.254 network.
- phyzome 3y agoA friend has reminded me that with Linux, it can be as basic as piping tar between the two machines using nc. (There's probably a friendly frontend to this somewhere that will allow the machines to mount each other's filesystems with fuse, though...)
- justinclift 3y ago> Turns out the fastest way to move data between 2 modern PCs ... With the caveat being that's only if you're not up for adding PCIe cards. If you are ok with adding some PCIe cards, then you can transfer things a lot faster than 1GB/s. :)
- kwanbix 3y agoCare to elaborate?
- hatefulmoron 3y agoI assume they're referring to getting a couple of NICs
- justinclift 3y agoNetwork cards that go a lot faster than 10GbE are common. They're widely used (with many different types) in IT data centres, home labs, and probably other places too. Heaps of them are on Ebay. As a random search just now for "Mellanox 25GbE" on US Ebay: * https://www.ebay.com/itm/134435757546 https://www.ebay.com/itm/134435757546 * https://www.ebay.com/itm/355348422765 https://www.ebay.com/itm/355348422765 (there are hundreds of individual results) Searching for "Mellanox 50GbE": * https://www.ebay.com/itm/225021493021 https://www.ebay.com/itm/225021493021 * https://www.ebay.com/itm/233915360659 https://www.ebay.com/itm/233915360659 (less results) There are older generation ones too, doing 40GbE: * https://www.ebay.com/itm/305046322527 https://www.ebay.com/itm/305046322527 * https://www.ebay.com/itm/166350081025 https://www.ebay.com/itm/166350081025 (hundreds of results again) With those older generation cards, some care is needed depending upon the OS being run. If you're using Linux you should be fine. If you're running some other OS though (eg ESXi) then they might have dropped out of the "supported list" for the OS and not have their drivers included.
- oso2k 3y agoI bought a bunch cheap HP 40Gbps NICs [0] when they were $13 but need PCI-E risers to make them fit a full height slot. Work fine in pfSense & Fedora. [0] https://www.ebay.com/itm/333682185870 https://www.ebay.com/itm/333682185870