5 ms·

Bandwidth doesn't multiply like that here, and we measured it rather than assumed it. A MoE layer needs 16 expert reads and can't proceed until the slowest one

Bandwidth doesn't multiply like that here, and we measured it rather than assumed it. A MoE layer needs 16 expert reads and can't proceed until the slowest one lands, so a layer costs the max over its reads, not the sum. Going from one drive to four (13.6 → ~33 GB/s of combined ceilings) took decode from ~52% to 100% of our number — not 4× — with Every drive already at 90–100% of its own ceiling. RAID-0 was one of the first things tried and it lost: striping makes every read touch every drive, so the slowest drive sets every barrier. What moves this is per-read latency and read scheduling, and for long prompts not re-reading each layer's experts eight times. Numbers in results/SCALING.md and results/PREFILL.md.

lowbloodsugar 9d ago
Would the 40 Mac’s work with pipelining though?
- adrian_b 9d ago
  No matter how many external drives you gather, the data coming from them must be squeezed through the peripheral interfaces of the Apple SoC. So your CPU, made by Apple, Intel, AMD etc., has a number of PCIe lanes and a number of USB/Thunderbolt ports for connecting peripherals. Those have an aggregated throughput, which sets an upper limit for the amount of data that can be read per second from all the peripheral devices. In a given computer, usually not all the lanes and ports of the CPU are actually connected, so the limit may be even lower. In desktop PCs and mini-PCs, usually only 4 + 4 = 8 PCIe lanes are available for SSDs, and when there are more SSD sockets they share some of those lanes. A much higher SSD throughput could be achieved in a desktop PC by using the GPU connector with an SSD adapter for M.2 SSDs, which has 16 PCIe 5.0 lanes, with a 64 GByte/s throughput. Taking out the GPU might actually be OK for doing AI inference, because a beefy CPU like a Ryzen 9950X should be able to keep up with a reading throughput of 88 GB/s from 6 SSDs (2 on the motherboard and 4 on the add-on PCIe card), while computing inference in the INT8 or BF16 formats, so the absence of the GPU would not reduce the inference speed when it is limited by the speed of reading the weights.

lowbloodsugar 9d ago

Would the 40 Mac’s work with pipelining though?

adrian_b 9d ago
No matter how many external drives you gather, the data coming from them must be squeezed through the peripheral interfaces of the Apple SoC. So your CPU, made by Apple, Intel, AMD etc., has a number of PCIe lanes and a number of USB/Thunderbolt ports for connecting peripherals. Those have an aggregated throughput, which sets an upper limit for the amount of data that can be read per second from all the peripheral devices. In a given computer, usually not all the lanes and ports of the CPU are actually connected, so the limit may be even lower. In desktop PCs and mini-PCs, usually only 4 + 4 = 8 PCIe lanes are available for SSDs, and when there are more SSD sockets they share some of those lanes. A much higher SSD throughput could be achieved in a desktop PC by using the GPU connector with an SSD adapter for M.2 SSDs, which has 16 PCIe 5.0 lanes, with a 64 GByte/s throughput. Taking out the GPU might actually be OK for doing AI inference, because a beefy CPU like a Ryzen 9950X should be able to keep up with a reading throughput of 88 GB/s from 6 SSDs (2 on the motherboard and 4 on the add-on PCIe card), while computing inference in the INT8 or BF16 formats, so the absence of the GPU would not reduce the inference speed when it is limited by the speed of reading the weights.