6 ms·
> SSD streaming to GPU Is this solution based on what Apple describes in their 2023 paper 'LLM in a flash' [1]? 1: https://arxiv.org/abs/2312.11514 https://ar
by firstbabylonian 6mo ago
> SSD streaming to GPU
Is this solution based on what Apple describes in their 2023 paper 'LLM in a flash' [1]?
1: https://arxiv.org/abs/2312.11514 https://arxiv.org/abs/2312.11514
- simonw 6mo agoYes. I collected some details here: https://simonwillison.net/2026/Mar/18/llm-in-a-flash/ https://simonwillison.net/2026/Mar/18/llm-in-a-flash/
- superjan 6mo agoThat was a very good summary. One detail the post could use is mentioning that 4 or 10 experts invoked where selected from the 512 experts the model has per layer (to give an idea of the savings).
- anemll 6mo agoThanks for posting this, that's how I first found out about Dan's experiment! SSD speed doubled in the M5P/M generation, that makes it usable! I think one paper under the radar is "KV Prediction for Improved Time to First Token" https://arxiv.org/abs/2410.08391 https://arxiv.org/abs/2410.08391 which hopefully can help with prefill for Flash streaming.
- Yukonv 6mo agoThat’s exactly what I thought about. Getting my hands on an M5 Max this week and going to see hows Dan’s experiment performs with faster I/O. Also going to experiment with running active parameters at Q6 or Q8 since output is I/O bottlenecked there should room for higher accuracy compute.
- 3abiton 6mo agoTo be fair, it's "possible" to run such setup with llama.cpp with ssd offload. It's just abysmal TG speeds. But it's possible.
- anemll 6mo agoCheck my repo, I had added some support for GUFF/untloth, Q3,Q5/Q8 https://github.com/Anemll/flash-moe/blob/iOS-App/docs/gguf-hybrid-bringup-log.md https://github.com/Anemll/flash-moe/blob/iOS-App/docs/gguf-h...
- trebligdivad 6mo agoI guess this is all set up to show off the new high-bandwidth-flash stuff that's due out soon?
- zozbot234 6mo agoA similar approach was recently featured here: https://news.ycombinator.com/item?id=47476422 https://news.ycombinator.com/item?id=47476422 Though iPhone Pro has very limited RAM (12GB total) which you still need for the active part of the model. (Unless you want to use Intel Optane wearout-resistant storage, but that was power hungry and thus unsuitable to a mobile device.)
- simonw 6mo agoYeah, this new post is a continuation of that work.
- Aurornis 6mo ago> Though iPhone Pro has very limited RAM (12GB total) which you still need for the active part of the model. This is why mixture of experts (MoE) models are favored for these demos: Only a portion of the weights are active for each token.
- zozbot234 6mo agoYes but most people are still running MoE models with all experts loaded in RAM! This experiment shows quite clearly that some experts are only rarely needed, so you do benefit from not caching every single expert-layer in RAM at all times.
- jnovek 6mo agoI’m so confused in these comments right now — I thought you had to load an entire MoE model and sparseness just made it so you can traverse the model more quickly.
- Aurornis 6mo agoThat's not what this test shows. It's just loading the parts of the model that are used in an on-demand fashion from flash. The iPhone 17 Pro only has 12GB of RAM. This is a -17B MoE model. Even quantized, you can only realistically fit one expert in RAM at a time. Maybe 2 with extreme quantization. It's just swapping them out constantly. If some of the experts were unused then you could distill them away. This has been tried! You can find reduced MoE models that strip away some of the experts, though it's ony a small number. Their output is not good. You really need all of the experts to get the model's quality.
- foobiekr 6mo agoThis is not entirely dissimilar to what Cerebus does with their weights streaming.
- manmal 6mo agoAnd IIRC the Unreal Engine Matrix demo for PS5 was streaming textures directly from SSD to the engine as well?
- WatchDog 6mo agoYeah, also "RTX IO", and Microsoft "DirectStorage". What was more interesting about the unreal engine demo, was that they can stream not only textures, but geometry too. Virtual texturing had been around a long time, but virtual geometry with nanite is really interesting.