5 ms·

SSD streaming on an M5 Max 128GB: https://x.com/antirez/status/2082136334160818528 https://x.com/antirez/status/2082136334160818528 Soon decent speed across tw

by antirez 2mo ago

SSD streaming on an M5 Max 128GB: https://x.com/antirez/status/2082136334160818528 https://x.com/antirez/status/2082136334160818528 Soon decent speed across two Mac Studios with 512GB of RAM.

tito 2mo ago
wow cool! I like watching the new models come out and how they end up crammed in to run on local machines. I learned what mxfp4 is thanks to this latest Kimi release - although it sounds like it means that there's less room for compression in the model compared to others.
xfour 2mo ago
Cool stuff. Do you have a the hardware and a way to bridge the compute? Or just hopeful?
- weikju 2mo ago
  Supposedly RDMA over Thunderbolt 5 can allow this.
whatsThisBtn4 2mo ago
0.3tx per second is decent speed? And it gets worse with every token.
- smallerize 2mo ago
  That's not what he said. He said with 2x the hardware it will be faster.
- zozbot234 2mo ago
  That's not what parent said, but this is already quite decent speed for unattended inference (overnight or even spanning multiple business days) which is arguably the right target for this model. This is a challenging model to infer locally, it has roughly ~115 GB of dense active parameters(!) plus ~25 GB of sparsely routed experts per token. Plus the KV cache (which is actually reasonably lean for this one model, around 27GB for a full 1Mtok context). What you're seeing in antirez's video is essentially the performance we should expect from a 128GiB system that has to load sparsely routed experts in full from disk because there's no real room for caching them. 192GiB Gorgon Halo systems will be an interesting future target for this model, the best you can do with 128GiB or less is probably to push batching higher in order to amortize the weights traffic over multiple inferences - which of course will sink single-session speeds even lower for a modest gain in total throughput.
  - whatsThisBtn4 2mo ago
    Would it even finish over a few weeks? For AI agents this would take a year
  - searealist 2mo ago
    No one will ever derive any utility from running models at this speed. Please prove me wrong. Give me the number of tokens input and output (and dont forget about reasoning) and acceptable time to wait for it and the use case.
    - dannyw 2mo ago
      The OpenAI and Anthropic batch APIs give you a 50% discount. This is like 98%.
      - searealist 2mo ago
        I'm not aware of any service that gives you a 98% discount and is served off of M1s. Did you do the math for what this would cost vs K3 on openrouter?
        dannyw 2mo ago
        I just mean in terms of incremental inference cost. If you already committed to your hardware, huge models at single or sub-digit TPS are still useful. LLM-as-judge is a good use case. If your machine is gonna be idle overnight (and the power efficiency is excellent here), why pay openrouter if it’s not an interactive workload?
        searealist 2mo ago
        Because it's probably still more expensive in electricity costs compared to openrouter. Certainly it's not 98% cheaper.
        zozbot234 2mo ago
        antirez's own video (the one I referenced in my comment) shows K3 inference running on M5 Max, not M1-series silicon (which is OP). M1 series has far lower SSD read throughput and memory bandwidth, and can barely fit the model weights on its maxed out internal storage (2TB). (This is why the linked OP resorts to streaming the sparse parameters from the network which is incredibly slow.)
api 2mo ago
I’ve wondered for a while: given the lower cost of SSD per GB could you build a very wide RAID0 style striped array of SSDs (maybe one per slot) to get almost RAM like read speeds? To really go fast you’d probably have to do PCB layout and do like 256 or 1024 chips in parallel with a fast SRAM aggregation buffer feeding a GPU or TPU rig. Or could you do the same with custom layout of cheap slower RAM? I wonder if anyone is doing this? You would flash in a model and then just run it. It would need RAM for context but much less of it.
- colingauvin 2mo ago
  The problem is you'd have to traverse the PCI-E bus every transfer. Even with DMA it still has to physically get off the drive and onto the card. Even with the mythical PCI-E 6, if you had enough NVMEs to saturate, you have to do two transfers to get it to the inference hardware. And that tops out at 128 GB/s which is roughly the speed of DDR5 but with one extra hop. Optane would actually be useful in this era. Intel was ahead of their time.
  - api 2mo ago
    That’s why I think you’d have to build boards with raw flash chips and do your own bus to make it rival fast RAM.
    - colingauvin 2mo ago
      I'm guessing that there are storage vendors that already do that - maybe like Liqid. But then you've got to traverse the NIC as well.
chrisringrose 2mo ago
Thanks for sharing antirez! (huge fan btw) We've made some updates to deltafin since this was shared 5 days ago. I wonder if you'd be able to get better speeds now? https://github.com/gavamedia/deltafin https://github.com/gavamedia/deltafin

by antirez 2mo ago

SSD streaming on an M5 Max 128GB: https://x.com/antirez/status/2082136334160818528 https://x.com/antirez/status/2082136334160818528 Soon decent speed across two Mac Studios with 512GB of RAM.

tito 2mo ago

wow cool! I like watching the new models come out and how they end up crammed in to run on local machines. I learned what mxfp4 is thanks to this latest Kimi release - although it sounds like it means that there's less room for compression in the model compared to others.

xfour 2mo ago

Cool stuff. Do you have a the hardware and a way to bridge the compute? Or just hopeful?

weikju 2mo ago
Supposedly RDMA over Thunderbolt 5 can allow this.

whatsThisBtn4 2mo ago

0.3tx per second is decent speed? And it gets worse with every token.

smallerize 2mo ago
That's not what he said. He said with 2x the hardware it will be faster.
zozbot234 2mo ago
That's not what parent said, but this is already quite decent speed for unattended inference (overnight or even spanning multiple business days) which is arguably the right target for this model. This is a challenging model to infer locally, it has roughly ~115 GB of dense active parameters(!) plus ~25 GB of sparsely routed experts per token. Plus the KV cache (which is actually reasonably lean for this one model, around 27GB for a full 1Mtok context). What you're seeing in antirez's video is essentially the performance we should expect from a 128GiB system that has to load sparsely routed experts in full from disk because there's no real room for caching them. 192GiB Gorgon Halo systems will be an interesting future target for this model, the best you can do with 128GiB or less is probably to push batching higher in order to amortize the weights traffic over multiple inferences - which of course will sink single-session speeds even lower for a modest gain in total throughput.
- whatsThisBtn4 2mo ago
  Would it even finish over a few weeks? For AI agents this would take a year
- searealist 2mo ago
  No one will ever derive any utility from running models at this speed. Please prove me wrong. Give me the number of tokens input and output (and dont forget about reasoning) and acceptable time to wait for it and the use case.
  - dannyw 2mo ago
    The OpenAI and Anthropic batch APIs give you a 50% discount. This is like 98%.
    - searealist 2mo ago
      I'm not aware of any service that gives you a 98% discount and is served off of M1s. Did you do the math for what this would cost vs K3 on openrouter?
      - dannyw 2mo ago
        I just mean in terms of incremental inference cost. If you already committed to your hardware, huge models at single or sub-digit TPS are still useful. LLM-as-judge is a good use case. If your machine is gonna be idle overnight (and the power efficiency is excellent here), why pay openrouter if it’s not an interactive workload?
        searealist 2mo ago
        Because it's probably still more expensive in electricity costs compared to openrouter. Certainly it's not 98% cheaper.
      - zozbot234 2mo ago
        antirez's own video (the one I referenced in my comment) shows K3 inference running on M5 Max, not M1-series silicon (which is OP). M1 series has far lower SSD read throughput and memory bandwidth, and can barely fit the model weights on its maxed out internal storage (2TB). (This is why the linked OP resorts to streaming the sparse parameters from the network which is incredibly slow.)

api 2mo ago

I’ve wondered for a while: given the lower cost of SSD per GB could you build a very wide RAID0 style striped array of SSDs (maybe one per slot) to get almost RAM like read speeds? To really go fast you’d probably have to do PCB layout and do like 256 or 1024 chips in parallel with a fast SRAM aggregation buffer feeding a GPU or TPU rig. Or could you do the same with custom layout of cheap slower RAM? I wonder if anyone is doing this? You would flash in a model and then just run it. It would need RAM for context but much less of it.

colingauvin 2mo ago
The problem is you'd have to traverse the PCI-E bus every transfer. Even with DMA it still has to physically get off the drive and onto the card. Even with the mythical PCI-E 6, if you had enough NVMEs to saturate, you have to do two transfers to get it to the inference hardware. And that tops out at 128 GB/s which is roughly the speed of DDR5 but with one extra hop. Optane would actually be useful in this era. Intel was ahead of their time.
- api 2mo ago
  That’s why I think you’d have to build boards with raw flash chips and do your own bus to make it rival fast RAM.
  - colingauvin 2mo ago
    I'm guessing that there are storage vendors that already do that - maybe like Liqid. But then you've got to traverse the NIC as well.

chrisringrose 2mo ago

Thanks for sharing antirez! (huge fan btw) We've made some updates to deltafin since this was shared 5 days ago. I wonder if you'd be able to get better speeds now? https://github.com/gavamedia/deltafin https://github.com/gavamedia/deltafin