65 ms·

I wonder if faster SSDs would help? In particular you can still get used Optane SSDs on eBay, although they’re fairly pricey. (not the bogus m.2 ones that are

by pjdesno 10d ago

I wonder if faster SSDs would help? In particular you can still get used Optane SSDs on eBay, although they’re fairly pricey. (not the bogus m.2 ones that are slower than a halfway decent consumer NVMe)

Argonautlabs 10d ago
Probably not. Each read here is a whole 17.5 MB expert file, so the time per read is set by the drive's throughput, not its access latency — 17.5 MB at 7 GB/s is ~2.5 ms, which is what we measure at queue depth 1 on the SN8100s. What actually moves the barrier is how fast the slowest of 16 concurrent whole-file reads completes: more drives on direct ports, stable tail behaviour under load, and scheduling. Our ladder shows even that with diminishing returns (one drive ≈52% of four, three ≈90%).
- xtracto 10d ago
  Could RAID0 help?
  - adrian_b 10d ago
    When you have heterogeneous SSDs, e.g. you mix PCIe 5.0, PCIe 4.0 and Thunderbolt interfaces, you can obtain a greater throughput by managing in software the distribution of data, than by using RAID0. RAID0 works fine only when all the interfaces have the same speed. If one SSD is twice faster than the other, in order to achieve maximal throughput, you must take care to place the data in such a way so that you will need to read twice more data from the twice faster SSD. In general, you must distribute the data so that the amounts read from each SSD are proportional with the throughputs of the SSDs. One could write a modified RAID0 device driver, which would use unequal stripes, with widths proportional with the SSD throughputs, but I am not aware of any such already existing RAID0 driver.
    - zamadatix 9d ago
      One approach I've seen used is: mdadm --create /dev/md0 --level=0 --raid-devices=3 /dev/nvme0n1p1 /dev/nvme0n1p2 /dev/nvme1n1p1 Which is effectively a way to get any positive integer m:n ratioed bandwidth distribution over any number of any sized drives.
      - adrian_b 9d ago
        This seems to be an acceptable solution. This would not work with HDDs, because the time to seek between 2 partitions of the same device would cause abysmal performance, but on SSDs this should work fine, even if some SSD controllers might still have a lower throughput when reading non-sequential pages.
zamadatix 10d ago
LLMs tend to care for sequential bandwidth more than random access. From that perspective, Optane probably doesn't offer much of an advantage vs other NVMe drives even from the same era while losing out greatly to newer drives with faster busses to fill. There could be something special to this particular instance but https://github.com/argonautlabsai/deltafin/blob/main/k3-public-bench/results/SCALING.md https://github.com/argonautlabsai/deltafin/blob/main/k3-publ... and the overall tok/s numbers seem to suggest it's just the sequential that's playing in.
adrian_b 10d ago
Most desktops and mini-PCs can read simultaneously from only 2 internal M.2 SSDs, 1 of which is PCIe 5.0 and the other is PCIe 4.0. They may have more M.2 sockets but usually except for the first all the others share the same PCIe 4.0 path. Those 2 M.2 SSDs have a theoretical maximum aggregated throughput of 24 GByte/s, but what is achievable in practice is typically only slightly above 20 GB/s. If you have 1 or 2 USB 4 / Thunderbolt 4 ports, the aggregated throughput can be increased with external SSDs, like in TFA. With 2 internal SSDs and 2 fast external SSDs, a higher throughput than in TFA is possible.

by pjdesno 10d ago

Argonautlabs 10d ago

Probably not. Each read here is a whole 17.5 MB expert file, so the time per read is set by the drive's throughput, not its access latency — 17.5 MB at 7 GB/s is ~2.5 ms, which is what we measure at queue depth 1 on the SN8100s. What actually moves the barrier is how fast the slowest of 16 concurrent whole-file reads completes: more drives on direct ports, stable tail behaviour under load, and scheduling. Our ladder shows even that with diminishing returns (one drive ≈52% of four, three ≈90%).

xtracto 10d ago
Could RAID0 help?
- adrian_b 10d ago
  When you have heterogeneous SSDs, e.g. you mix PCIe 5.0, PCIe 4.0 and Thunderbolt interfaces, you can obtain a greater throughput by managing in software the distribution of data, than by using RAID0. RAID0 works fine only when all the interfaces have the same speed. If one SSD is twice faster than the other, in order to achieve maximal throughput, you must take care to place the data in such a way so that you will need to read twice more data from the twice faster SSD. In general, you must distribute the data so that the amounts read from each SSD are proportional with the throughputs of the SSDs. One could write a modified RAID0 device driver, which would use unequal stripes, with widths proportional with the SSD throughputs, but I am not aware of any such already existing RAID0 driver.
  - zamadatix 9d ago
    One approach I've seen used is: mdadm --create /dev/md0 --level=0 --raid-devices=3 /dev/nvme0n1p1 /dev/nvme0n1p2 /dev/nvme1n1p1 Which is effectively a way to get any positive integer m:n ratioed bandwidth distribution over any number of any sized drives.
    - adrian_b 9d ago
      This seems to be an acceptable solution. This would not work with HDDs, because the time to seek between 2 partitions of the same device would cause abysmal performance, but on SSDs this should work fine, even if some SSD controllers might still have a lower throughput when reading non-sequential pages.

zamadatix 10d ago

LLMs tend to care for sequential bandwidth more than random access. From that perspective, Optane probably doesn't offer much of an advantage vs other NVMe drives even from the same era while losing out greatly to newer drives with faster busses to fill. There could be something special to this particular instance but https://github.com/argonautlabsai/deltafin/blob/main/k3-public-bench/results/SCALING.md https://github.com/argonautlabsai/deltafin/blob/main/k3-publ... and the overall tok/s numbers seem to suggest it's just the sequential that's playing in.

adrian_b 10d ago

Most desktops and mini-PCs can read simultaneously from only 2 internal M.2 SSDs, 1 of which is PCIe 5.0 and the other is PCIe 4.0. They may have more M.2 sockets but usually except for the first all the others share the same PCIe 4.0 path. Those 2 M.2 SSDs have a theoretical maximum aggregated throughput of 24 GByte/s, but what is achievable in practice is typically only slightly above 20 GB/s. If you have 1 or 2 USB 4 / Thunderbolt 4 ports, the aggregated throughput can be increased with external SSDs, like in TFA. With 2 internal SSDs and 2 fast external SSDs, a higher throughput than in TFA is possible.