8 ms·

Diskomator – NVMe-TCP at your fingertips

141 points by simjue 3y ago

anotherhue 3y ago
>I'd like to live to see a future where people build appliances like this for various purposes, not just this specific NVMe one. For example, a nice thing to have would be an appliance whose only job is to make all local displays available via Miracast. I hope this repository is inspiration enough for an interested soul, to get this off the ground. Very nice idea.
- deleted 3y ago
  [deleted]
mike_d 3y ago
I seem to recall an announcement from Western Digital(?) years ago about a line of hard drives with a direct ethernet interface. Does anyone remember the same or what might have come of it? The market is saturated with solutions for middle-boxes that make hard drives talk to networks, but nobody seems to be directly addressing the problem of we just want storage network accessible.
- _zoltan_ 3y ago
  Kioxia has a line of products (EM6) that has an embedded Marvell NVMeoF/{TCP,RDMA} controller and Foxconn has a chassis that can take them and expose them just directly on the network. neat stuff.
  - gorkish 3y ago
    Neat if you can stomach paying for the novelty. Let's see an open product that does the same for any old M.2 stick
    - _zoltan_ 3y ago
      SPDK + RDMA works just fine :)
      - gorkish 3y ago
        Yeah i know the software exists; it's finding ways to actually build systems from it that sucks. 2.5 gigabit ethernet has absolutely killed dead all momentum for getting faster interfaces in consumer hardware and SBCs unfortunately. I don't want to build a 2-drive OSD using a comparatively gargantuan microATX motherboard.
- toast0 3y ago
  > The market is saturated with solutions for middle-boxes that make hard drives talk to networks, but nobody seems to be directly addressing the problem of we just want storage network accessible. Most hard drives run a serial console on some of the jumpers. You can easily run PPP or SLIP over that. QED :P
  - ronsor 3y ago
    Nothing like those blazing fast 115kbps read/write speeds.
  - csdvrx 3y ago
    Tell me more, like a brand and model where I can do that! BTW if you already explored that, would you know how to alter the SLC/MLC ratio by any chance? Modern QLC drive often have a SLC area for buffering. With the right firmware tools, it should be possible to take a 4 TB QLC drive and convert it to a 1 SLC drive to get more performance.
    - EvanAnderson 3y ago
      An older reference for hacking hard disk drives: https://spritesmods.com/?art=hddhack https://spritesmods.com/?art=hddhack There were some firmware bugs on Seagate Barracuda SATA drives that could be worked-around w/ serial console. I don't remember the specifics though. If you some search-engining on hard drive manufacturers and "serial console" you'll find indexed pages. (Presumably the really interesting stuff is buried deep down in forum posts, etc.) Just doing a couple quick searches got me some stuff.
- gorkish 3y ago
  > The market is saturated with solutions for middle-boxes that make hard drives talk to networks It's my experience that these boxes try to do a hell of a lot more than just putting drives-on-network and that is why they all suck and are expensive. The NVMe-oF fabric devices out there all seem to command a ridiculous premium when the reality is they ought to be very simple and easily cost-optimized.
- matja 3y ago
  There was a line of products called EtherDrive made by Coraid in the early 2000's that was basically a SATA to ATAoE bridge on the most basic devices, up to rack-mount solutions that used a Linux OS and Dell HBA to run vblade (https://github.com/OpenAoE/vblade https://github.com/OpenAoE/vblade) to expose a Linux MD array carved up using LVM2. ATAoE ("aoe" in the Linux kernel) is nice because it is very lightweight in both terms of code to implement it (~2-3kloc, basically just stuff an ATA packet in an ethernet frame), low network overhead, and ease to setup (no IP addresses).
- wkennington 3y ago
  Seagate kinetic https://www.seagate.com/files/www-content/product-content/hdd-fam/kinetic-hdd/en-us/docs/kinetic-ds1835-1-1110us.pdf https://www.seagate.com/files/www-content/product-content/hd...
- wmf 3y ago
  I don't think the economics of this work because an Ethernet network is probably much more expensive than SAS and network configuration is more complex (you could make it zeroconf but that's also zero security).
  - mschuster91 3y ago
    > I don't think the economics of this work because an Ethernet network is probably much more expensive than SAS Well, iSCSI has been around for ages, and because people got fed up with Fibre Channel requiring dedicated switches and transceiver components, first came FCoE that allowed using regular network transceivers and switches and then FCIP/iFCP that added regular IP routing to the mix but never saw much uptake.
    - justinclift 3y ago
      > first came FCoE that allowed using regular network [stuff] From rough memory, didn't FCoE start out that way only in theory? With the reality that people needed to buy FCoE rated equipment, which was priced at "enterprise pricing" levels. Pretty sure that was FCoE, and I'm not mis-remembering that from something else... :)
- justsomehnguy 3y ago
  > Does anyone remember the same or what might have come of it? Nothing, because it makes each disk quite costly and by the 2014 nobody wanted costly and slow HDDs. Check the Seagate offering up there it has 1Gbit interface. You can't even run the drive at full sequential read/write speed over it. And having a two 10Gbit ports on each drive would require having two 10Gbit switches, which by 2014 were still quite costly. EM6 solution[0,1] is neat but at least it both the quite packed both by the spec and the price but delivers a lot of IOPS and throughput. [0] https://www.ingrasys.com/assets/files/Datasheet_ES2000_20211026.pdf https://www.ingrasys.com/assets/files/Datasheet_ES2000_20211... [1] https://www.servethehome.com/ethernet-ssds-hands-on-with-the-kioxia-em6-nvmeof-ssd/ https://www.servethehome.com/ethernet-ssds-hands-on-with-the...
- cyberge99 3y ago
  I believe you are talking about the WD EX2 boxes. NAS in a box.
withinboredom 3y ago
I'm looking forward to Longhorn[1] taking advantage of this technology. [1]: https://github.com/longhorn/longhorn https://github.com/longhorn/longhorn
EnigmaCurry 3y ago
How does this compare with iSCSI?
- withinboredom 3y ago
  Like an iops improvement of 30%+ and latency improvement of 20%+[1], ish. [1]: https://www.reddit.com/r/Proxmox/comments/134kqy3/iscsi_and_nvmetcp_shared_storage_comparison/ https://www.reddit.com/r/Proxmox/comments/134kqy3/iscsi_and_...
  - justinclift 3y ago
    There's no way to combine the NVMe drives into a larger sized unit for redundancy / failover though, so not sure what kind of future uptake this could have.
    - withinboredom 3y ago
      I’m not sure what you mean. You can add the disks to a software RAID in the worst case. Are you talking about on the host?
      - justinclift 3y ago
        Yeah. It seems like directly presenting raw disks to the network means any kind of redundancy would need to be done by whatever device/host/thing is mounting the storage. And doing that over the network (instead of over a local PCIe bus) seems like it'll have some trade-offs. :/
    - aseipp 3y ago
      Everyone who uses NVMe-over-network-transport simply does redundancy at the client layer. The networking gear is very robust, and it is easier to optimize the "data plane" path this way (map storage queues <-> network queues) so the actual storage system does less work, which improves cost and density. That also means clients can have their own redundancy solutions that more closely match their requirements, e.g. filesystems can use block devices and implement RAID10 for e.g. virtual machine storage, while userspace applications may use them directly with Reed-Solomon(14,10) and manage the underlying multiple block devices themselves. This all effectively improves density and storage utilization even further. NVMe-over-network (fabrics w/ RDMA, TCP, ROCEv2) is very popular for doing disaggregated storage/compute, and things like Nvidia Bluefield push the whole thing down into networking cards on the host so you don't even see the "over network" part. You have a diskless server, plug in some Bluefield cards, and it exposes a bunch of NVMe drives to the host, as if they were plugged in physically. That makes it much easier to scale compute and storage separately (and also effectively increases the capacity of the host machine since it no longer is using up bandwidth and CPU on those tasks.)
gorkish 3y ago
This is awesome. Would be exciting if it can be extended to support NVMe-oF as well with RDMA via RoCEv2. A SBC running something like this with at least 2x10GbE and two M.2 slots and 2 sata ports would be an absolute dream device for me.
- justinclift 3y ago
  Having the devices exposed "raw" means there's no redundancy on the device though. :(
  - gorkish 3y ago
    That’s exactly what I want. I want it to be cost optimized enough to just use a bunch
EvanAnderson 3y ago
I never really used Mac machines, but I always appreciated "target disk mode". This sounds similar, albeit over a network (which could simply be a straight-thru cable between Ethernet NICs on two machines). Edit: Yeah. I forgot about actually saying what "target disk mode was". There's a child post that mentions it, so I'll refrain. I will say that I saw it used in imaging computers in a college computer lab setting back in the early 2000's. I definitely wished my PCs could've done it. It looked like a very handy feature. Presumably it would make fixing OS boot issues easier, as well as just harvesting files off a machine that was otherwise not operating properly due to OS issues.
- metadat 3y ago
  Diskomater appears to be an open source, platform agnostic version of this underlying technology! In case, like me, "Target Disk Mode" is not something you're familiar with: > People also ask What is the target disk mode? If you have two Mac computers with USB, USB-C or Thunderbolt ports, you can connect them so that one of them appears as an external hard disk on the other. This is called target disk mode. Note: If either of the computers has macOS 11 or later installed, you must connect the two computers using a Thunderbolt cable. Sidenote: Thunderbolt cables are special USB-C cables, i.e. the wire between your Mac and the USB-C power brick.
  - cjdell 3y ago
    I remember using Target Disk Mode on my iBook G4 over FireWire cables! Very useful for copying large amounts of data before gigabit Ethernet was common place.
  - mschuster91 3y ago
    > Sidenote: Thunderbolt cables are special USB-C cables, i.e. the wire between your Mac and the USB-C power brick. Not exactly - that is only sort-of valid for Thunderbolt 3 and its successor/merger USB-C 4. TB1 and 2 used the Mini DisplayPort connector. TB3/4 use the USB-C connector, but unlike USB-C, at longer lengths (per standard: 0.5m [1], but there longer passive cables as well, e.g. [2]) are the cable isn't passive or semi-passive (cables supporting USB-C PD for higher currents/voltages contain "marker" chipsets in the plugs) - it's an "active" cable with special transmission/driver circuitry to achieve tolerable signal integrity. [1] https://en.wikipedia.org/wiki/Thunderbolt_(interface)#Thunderbolt_3 https://en.wikipedia.org/wiki/Thunderbolt_(interface)#Thunde... [2] https://www.amazon.de/Cable-Matters-Zertifiziertes-Thunderbolt-unterst%C3%BCtzt-Schwarz/dp/B07SH2B2B8?th=1 https://www.amazon.de/Cable-Matters-Zertifiziertes-Thunderbo...
jauntywundrkind 3y ago
Can't wait till there's a NVMe-QUIC.
londons_explore 3y ago
I think a far smaller version of this could be built on top of UEFI functionality... Ie. use UEFI to read/write the disks. UEFI to send/receive packets. UEFI to draw a splash image onto the screen. Now, you don't need any network drivers, graphics drivers or disk/controller drivers.
- londons_explore 3y ago
  I kinda wish Linux would implement support for all the UEFI functionality. It would let you build micro systems which 'just worked' with no drivers. The downside is that UEFI drivers tend to be barebones and not high performance. But it is a nice fallback to know that you will always at least have something that works. For example, here is the API for disk access offered by UEFI: https://uefi.org/specs/UEFI/2.10/13_Protocols_Media_Access.html#block-i-o-2-protocol https://uefi.org/specs/UEFI/2.10/13_Protocols_Media_Access.h...
  - pzmarzly 3y ago
    This is already possible with kernel patches in https://github.com/osresearch/safeboot-loader https://github.com/osresearch/safeboot-loader . It is extremely cool, but the README there also explains why it's not necessarily a good idea.
- NovemberWhiskey 3y ago
  I mean, as a proof of concept, that would be neat. As a useful service; not so much with the non-optimization of the UEFI abstraction layer.
- vacuity 3y ago
  As someone who is completely ignorant about this, what does it mean to use UEFI? I was under the impression that UEFI is just the firmware interface presented at boot time to configure stuff. So is this a way to do complex actions at boot time?
gdgghhhhh 3y ago
I don't get it. Isn't this just a live CD that setups nvmt like described here? https://blogs.oracle.com/linux/post/nvme-over-tcp https://blogs.oracle.com/linux/post/nvme-over-tcp
- sheepscreek 3y ago
  I guess? But neatly packaged and pre-configured. I mean, technically, every Linux distribution is “just a(nother) live cd” with a slightly different configuration and some custom packages thrown in. To each their own, yeah?
londons_explore 3y ago
Does NVMe-TCP have any support on Windows? Windows supports iSCSI clients/servers... Isn't it easier to emulate that and then you have a much wider range of possible clients?
- deleted 3y ago
  [deleted]
gigel82 3y ago
Now we just need a super cheap SBC with tons of PCIe lanes to chuck all those old M.2 drives into.
- sheepscreek 3y ago
  This has been my plan over the last few months! Finding a cheap PCIe 4 motherboard + CPU with many slots has been a challenge. Need PCIe 4 or else you’ll quickly saturate your bandwidth. Even with PCIe 4 - you still can. The fastest (affordable) network solution means using 10 GbE - which can be saturated by just one modern gen NVMe SSD :-/
  - justinclift 3y ago
    > The fastest (affordable) network solution means using 10 GbE ... That's one of those "it depends" things. If you only have a small number of computers that need connecting, and you're ok with using 2nd hand gear... then 40GBe Mellanox adapters from Ebay are pretty affordable. eg stuff like this (there's a bunch): * https://www.ebay.com.au/itm/293393033570 https://www.ebay.com.au/itm/293393033570 * https://www.ebay.com.au/itm/175163685084 https://www.ebay.com.au/itm/175163685084 Note - I don't know those sellers at all.
- wmf 3y ago
  There's the Asustor although it only has one lane for each SSD.
neverartful 3y ago
Next thing you know someone is going to want to run a 'disklet' (probably written in Java) within the data path of these devices. Shudder!
askvictor 3y ago
Would be nice to add this to Ventoy
- theblazehen 3y ago
  AFAIK ventoy can boot EFI files
jmprspret 3y ago
Very personal opinion, but I think the image is overcomplicated. Fedora base + systemd + sshd + application? This can surely be smushed down to being a go-krazy image. I guess then, you'd have to rewrite in go, and device support would be an issue
- m463 3y ago
  He wrote systemd, so it might be simpler for him.
  - jmprspret 3y ago
    I'm aware. :)
- jon-wood 3y ago
  There's more files yes, but I don't agree that it's more complicated because those files come together in the form of some well understood components that almost anyone who's done a bit of Linux admin understands to a reasonable degree, whereas your hypothetical Go application is going to end up being a completely custom implementation of NVMe-TCP, and if there's any problems with it you're on your own.
bitinerant 3y ago
I'm missing half of the picture. How would one mount a disk on the other end of the TCP connection?
- deleted 3y ago
  [deleted]
- bantunes 3y ago
  With the `nvme` CLI tool https://www.linuxjournal.com/content/data-flash-part-iii-nvme-over-fabrics-using-tcp https://www.linuxjournal.com/content/data-flash-part-iii-nvm...
Alifatisk 3y ago
What’s the idea with the bacteria emoji?
replete 3y ago
Interesting project. Slightly related is Ventoy [0]. Install Ventoy onto a USB disk drive and it will create a bootable partition that can mount your Ibootable images (including ISOs) onto your baremetal from the second partition it creates. In effect you can just load up a USB drive with ISOs and install onto baremetal from them. Super handy for distro hoppers and appealing if you don't want to fart around with network boot but just want to install something on a computer. I was trying to install Windows 11 and just wondered if there was an EFI thing that could just mount my USB, and Ventoy exists and works pretty well. I actually couldn't install windows without it on one system, just didn't like something about my installer media... [0]:https://www.ventoy.net/en/index.html https://www.ventoy.net/en/index.html
yencabulator 3y ago
The NVMe network server is part of systemd? Umm.. is that really relevant for an init replacement? https://www.freedesktop.org/software/systemd/man/latest/systemd-storagetm.html https://www.freedesktop.org/software/systemd/man/latest/syst...

141 points by simjue 3y ago

>I'd like to live to see a future where people build appliances like this for various purposes, not just this specific NVMe one. For example, a nice thing to have would be an appliance whose only job is to make all local displays available via Miracast. I hope this repository is inspiration enough for an interested soul, to get this off the ground. Very nice idea.

deleted 3y ago
[deleted]

mike_d 3y ago

I seem to recall an announcement from Western Digital(?) years ago about a line of hard drives with a direct ethernet interface. Does anyone remember the same or what might have come of it? The market is saturated with solutions for middle-boxes that make hard drives talk to networks, but nobody seems to be directly addressing the problem of we just want storage network accessible.

_zoltan_ 3y ago
Kioxia has a line of products (EM6) that has an embedded Marvell NVMeoF/{TCP,RDMA} controller and Foxconn has a chassis that can take them and expose them just directly on the network. neat stuff.
- gorkish 3y ago
  Neat if you can stomach paying for the novelty. Let's see an open product that does the same for any old M.2 stick
  - _zoltan_ 3y ago
    SPDK + RDMA works just fine :)
    - gorkish 3y ago
      Yeah i know the software exists; it's finding ways to actually build systems from it that sucks. 2.5 gigabit ethernet has absolutely killed dead all momentum for getting faster interfaces in consumer hardware and SBCs unfortunately. I don't want to build a 2-drive OSD using a comparatively gargantuan microATX motherboard.
toast0 3y ago
> The market is saturated with solutions for middle-boxes that make hard drives talk to networks, but nobody seems to be directly addressing the problem of we just want storage network accessible. Most hard drives run a serial console on some of the jumpers. You can easily run PPP or SLIP over that. QED :P
- ronsor 3y ago
  Nothing like those blazing fast 115kbps read/write speeds.
- csdvrx 3y ago
  Tell me more, like a brand and model where I can do that! BTW if you already explored that, would you know how to alter the SLC/MLC ratio by any chance? Modern QLC drive often have a SLC area for buffering. With the right firmware tools, it should be possible to take a 4 TB QLC drive and convert it to a 1 SLC drive to get more performance.
  - EvanAnderson 3y ago
    An older reference for hacking hard disk drives: https://spritesmods.com/?art=hddhack https://spritesmods.com/?art=hddhack There were some firmware bugs on Seagate Barracuda SATA drives that could be worked-around w/ serial console. I don't remember the specifics though. If you some search-engining on hard drive manufacturers and "serial console" you'll find indexed pages. (Presumably the really interesting stuff is buried deep down in forum posts, etc.) Just doing a couple quick searches got me some stuff.
gorkish 3y ago
> The market is saturated with solutions for middle-boxes that make hard drives talk to networks It's my experience that these boxes try to do a hell of a lot more than just putting drives-on-network and that is why they all suck and are expensive. The NVMe-oF fabric devices out there all seem to command a ridiculous premium when the reality is they ought to be very simple and easily cost-optimized.
matja 3y ago
There was a line of products called EtherDrive made by Coraid in the early 2000's that was basically a SATA to ATAoE bridge on the most basic devices, up to rack-mount solutions that used a Linux OS and Dell HBA to run vblade (https://github.com/OpenAoE/vblade https://github.com/OpenAoE/vblade) to expose a Linux MD array carved up using LVM2. ATAoE ("aoe" in the Linux kernel) is nice because it is very lightweight in both terms of code to implement it (~2-3kloc, basically just stuff an ATA packet in an ethernet frame), low network overhead, and ease to setup (no IP addresses).
wkennington 3y ago
Seagate kinetic https://www.seagate.com/files/www-content/product-content/hdd-fam/kinetic-hdd/en-us/docs/kinetic-ds1835-1-1110us.pdf https://www.seagate.com/files/www-content/product-content/hd...
wmf 3y ago
I don't think the economics of this work because an Ethernet network is probably much more expensive than SAS and network configuration is more complex (you could make it zeroconf but that's also zero security).
- mschuster91 3y ago
  > I don't think the economics of this work because an Ethernet network is probably much more expensive than SAS Well, iSCSI has been around for ages, and because people got fed up with Fibre Channel requiring dedicated switches and transceiver components, first came FCoE that allowed using regular network transceivers and switches and then FCIP/iFCP that added regular IP routing to the mix but never saw much uptake.
  - justinclift 3y ago
    > first came FCoE that allowed using regular network [stuff] From rough memory, didn't FCoE start out that way only in theory? With the reality that people needed to buy FCoE rated equipment, which was priced at "enterprise pricing" levels. Pretty sure that was FCoE, and I'm not mis-remembering that from something else... :)
justsomehnguy 3y ago
> Does anyone remember the same or what might have come of it? Nothing, because it makes each disk quite costly and by the 2014 nobody wanted costly and slow HDDs. Check the Seagate offering up there it has 1Gbit interface. You can't even run the drive at full sequential read/write speed over it. And having a two 10Gbit ports on each drive would require having two 10Gbit switches, which by 2014 were still quite costly. EM6 solution[0,1] is neat but at least it both the quite packed both by the spec and the price but delivers a lot of IOPS and throughput. [0] https://www.ingrasys.com/assets/files/Datasheet_ES2000_20211026.pdf https://www.ingrasys.com/assets/files/Datasheet_ES2000_20211... [1] https://www.servethehome.com/ethernet-ssds-hands-on-with-the-kioxia-em6-nvmeof-ssd/ https://www.servethehome.com/ethernet-ssds-hands-on-with-the...
cyberge99 3y ago
I believe you are talking about the WD EX2 boxes. NAS in a box.

withinboredom 3y ago

I'm looking forward to Longhorn[1] taking advantage of this technology. [1]: https://github.com/longhorn/longhorn https://github.com/longhorn/longhorn

EnigmaCurry 3y ago

How does this compare with iSCSI?

withinboredom 3y ago
Like an iops improvement of 30%+ and latency improvement of 20%+[1], ish. [1]: https://www.reddit.com/r/Proxmox/comments/134kqy3/iscsi_and_nvmetcp_shared_storage_comparison/ https://www.reddit.com/r/Proxmox/comments/134kqy3/iscsi_and_...
- justinclift 3y ago
  There's no way to combine the NVMe drives into a larger sized unit for redundancy / failover though, so not sure what kind of future uptake this could have.
  - withinboredom 3y ago
    I’m not sure what you mean. You can add the disks to a software RAID in the worst case. Are you talking about on the host?
    - justinclift 3y ago
      Yeah. It seems like directly presenting raw disks to the network means any kind of redundancy would need to be done by whatever device/host/thing is mounting the storage. And doing that over the network (instead of over a local PCIe bus) seems like it'll have some trade-offs. :/
  - aseipp 3y ago
    Everyone who uses NVMe-over-network-transport simply does redundancy at the client layer. The networking gear is very robust, and it is easier to optimize the "data plane" path this way (map storage queues <-> network queues) so the actual storage system does less work, which improves cost and density. That also means clients can have their own redundancy solutions that more closely match their requirements, e.g. filesystems can use block devices and implement RAID10 for e.g. virtual machine storage, while userspace applications may use them directly with Reed-Solomon(14,10) and manage the underlying multiple block devices themselves. This all effectively improves density and storage utilization even further. NVMe-over-network (fabrics w/ RDMA, TCP, ROCEv2) is very popular for doing disaggregated storage/compute, and things like Nvidia Bluefield push the whole thing down into networking cards on the host so you don't even see the "over network" part. You have a diskless server, plug in some Bluefield cards, and it exposes a bunch of NVMe drives to the host, as if they were plugged in physically. That makes it much easier to scale compute and storage separately (and also effectively increases the capacity of the host machine since it no longer is using up bandwidth and CPU on those tasks.)

gorkish 3y ago

This is awesome. Would be exciting if it can be extended to support NVMe-oF as well with RDMA via RoCEv2. A SBC running something like this with at least 2x10GbE and two M.2 slots and 2 sata ports would be an absolute dream device for me.

justinclift 3y ago
Having the devices exposed "raw" means there's no redundancy on the device though. :(
- gorkish 3y ago
  That’s exactly what I want. I want it to be cost optimized enough to just use a bunch

EvanAnderson 3y ago

I never really used Mac machines, but I always appreciated "target disk mode". This sounds similar, albeit over a network (which could simply be a straight-thru cable between Ethernet NICs on two machines). Edit: Yeah. I forgot about actually saying what "target disk mode was". There's a child post that mentions it, so I'll refrain. I will say that I saw it used in imaging computers in a college computer lab setting back in the early 2000's. I definitely wished my PCs could've done it. It looked like a very handy feature. Presumably it would make fixing OS boot issues easier, as well as just harvesting files off a machine that was otherwise not operating properly due to OS issues.

metadat 3y ago
Diskomater appears to be an open source, platform agnostic version of this underlying technology! In case, like me, "Target Disk Mode" is not something you're familiar with: > People also ask What is the target disk mode? If you have two Mac computers with USB, USB-C or Thunderbolt ports, you can connect them so that one of them appears as an external hard disk on the other. This is called target disk mode. Note: If either of the computers has macOS 11 or later installed, you must connect the two computers using a Thunderbolt cable. Sidenote: Thunderbolt cables are special USB-C cables, i.e. the wire between your Mac and the USB-C power brick.
- cjdell 3y ago
  I remember using Target Disk Mode on my iBook G4 over FireWire cables! Very useful for copying large amounts of data before gigabit Ethernet was common place.
- mschuster91 3y ago
  > Sidenote: Thunderbolt cables are special USB-C cables, i.e. the wire between your Mac and the USB-C power brick. Not exactly - that is only sort-of valid for Thunderbolt 3 and its successor/merger USB-C 4. TB1 and 2 used the Mini DisplayPort connector. TB3/4 use the USB-C connector, but unlike USB-C, at longer lengths (per standard: 0.5m [1], but there longer passive cables as well, e.g. [2]) are the cable isn't passive or semi-passive (cables supporting USB-C PD for higher currents/voltages contain "marker" chipsets in the plugs) - it's an "active" cable with special transmission/driver circuitry to achieve tolerable signal integrity. [1] https://en.wikipedia.org/wiki/Thunderbolt_(interface)#Thunderbolt_3 https://en.wikipedia.org/wiki/Thunderbolt_(interface)#Thunde... [2] https://www.amazon.de/Cable-Matters-Zertifiziertes-Thunderbolt-unterst%C3%BCtzt-Schwarz/dp/B07SH2B2B8?th=1 https://www.amazon.de/Cable-Matters-Zertifiziertes-Thunderbo...

jauntywundrkind 3y ago

Can't wait till there's a NVMe-QUIC.

londons_explore 3y ago

I think a far smaller version of this could be built on top of UEFI functionality... Ie. use UEFI to read/write the disks. UEFI to send/receive packets. UEFI to draw a splash image onto the screen. Now, you don't need any network drivers, graphics drivers or disk/controller drivers.

londons_explore 3y ago
I kinda wish Linux would implement support for all the UEFI functionality. It would let you build micro systems which 'just worked' with no drivers. The downside is that UEFI drivers tend to be barebones and not high performance. But it is a nice fallback to know that you will always at least have something that works. For example, here is the API for disk access offered by UEFI: https://uefi.org/specs/UEFI/2.10/13_Protocols_Media_Access.html#block-i-o-2-protocol https://uefi.org/specs/UEFI/2.10/13_Protocols_Media_Access.h...
- pzmarzly 3y ago
  This is already possible with kernel patches in https://github.com/osresearch/safeboot-loader https://github.com/osresearch/safeboot-loader . It is extremely cool, but the README there also explains why it's not necessarily a good idea.
NovemberWhiskey 3y ago
I mean, as a proof of concept, that would be neat. As a useful service; not so much with the non-optimization of the UEFI abstraction layer.
vacuity 3y ago
As someone who is completely ignorant about this, what does it mean to use UEFI? I was under the impression that UEFI is just the firmware interface presented at boot time to configure stuff. So is this a way to do complex actions at boot time?

gdgghhhhh 3y ago

I don't get it. Isn't this just a live CD that setups nvmt like described here? https://blogs.oracle.com/linux/post/nvme-over-tcp https://blogs.oracle.com/linux/post/nvme-over-tcp

sheepscreek 3y ago
I guess? But neatly packaged and pre-configured. I mean, technically, every Linux distribution is “just a(nother) live cd” with a slightly different configuration and some custom packages thrown in. To each their own, yeah?

londons_explore 3y ago

Does NVMe-TCP have any support on Windows? Windows supports iSCSI clients/servers... Isn't it easier to emulate that and then you have a much wider range of possible clients?

deleted 3y ago
[deleted]

gigel82 3y ago

Now we just need a super cheap SBC with tons of PCIe lanes to chuck all those old M.2 drives into.

sheepscreek 3y ago
This has been my plan over the last few months! Finding a cheap PCIe 4 motherboard + CPU with many slots has been a challenge. Need PCIe 4 or else you’ll quickly saturate your bandwidth. Even with PCIe 4 - you still can. The fastest (affordable) network solution means using 10 GbE - which can be saturated by just one modern gen NVMe SSD :-/
- justinclift 3y ago
  > The fastest (affordable) network solution means using 10 GbE ... That's one of those "it depends" things. If you only have a small number of computers that need connecting, and you're ok with using 2nd hand gear... then 40GBe Mellanox adapters from Ebay are pretty affordable. eg stuff like this (there's a bunch): * https://www.ebay.com.au/itm/293393033570 https://www.ebay.com.au/itm/293393033570 * https://www.ebay.com.au/itm/175163685084 https://www.ebay.com.au/itm/175163685084 Note - I don't know those sellers at all.
wmf 3y ago
There's the Asustor although it only has one lane for each SSD.

neverartful 3y ago

Next thing you know someone is going to want to run a 'disklet' (probably written in Java) within the data path of these devices. Shudder!

askvictor 3y ago

Would be nice to add this to Ventoy

theblazehen 3y ago
AFAIK ventoy can boot EFI files

jmprspret 3y ago

Very personal opinion, but I think the image is overcomplicated. Fedora base + systemd + sshd + application? This can surely be smushed down to being a go-krazy image. I guess then, you'd have to rewrite in go, and device support would be an issue

m463 3y ago
He wrote systemd, so it might be simpler for him.
- jmprspret 3y ago
  I'm aware. :)
jon-wood 3y ago
There's more files yes, but I don't agree that it's more complicated because those files come together in the form of some well understood components that almost anyone who's done a bit of Linux admin understands to a reasonable degree, whereas your hypothetical Go application is going to end up being a completely custom implementation of NVMe-TCP, and if there's any problems with it you're on your own.

bitinerant 3y ago

I'm missing half of the picture. How would one mount a disk on the other end of the TCP connection?

deleted 3y ago
[deleted]
bantunes 3y ago
With the `nvme` CLI tool https://www.linuxjournal.com/content/data-flash-part-iii-nvme-over-fabrics-using-tcp https://www.linuxjournal.com/content/data-flash-part-iii-nvm...

Alifatisk 3y ago

What’s the idea with the bacteria emoji?

replete 3y ago

Interesting project. Slightly related is Ventoy [0]. Install Ventoy onto a USB disk drive and it will create a bootable partition that can mount your Ibootable images (including ISOs) onto your baremetal from the second partition it creates. In effect you can just load up a USB drive with ISOs and install onto baremetal from them. Super handy for distro hoppers and appealing if you don't want to fart around with network boot but just want to install something on a computer. I was trying to install Windows 11 and just wondered if there was an EFI thing that could just mount my USB, and Ventoy exists and works pretty well. I actually couldn't install windows without it on one system, just didn't like something about my installer media... [0]:https://www.ventoy.net/en/index.html https://www.ventoy.net/en/index.html

yencabulator 3y ago

The NVMe network server is part of systemd? Umm.. is that really relevant for an init replacement? https://www.freedesktop.org/software/systemd/man/latest/systemd-storagetm.html https://www.freedesktop.org/software/systemd/man/latest/syst...