7 ms·
TurboQuant KV Compression and SSD Expert Streaming for M5 Pro and IOS
- boogerlad 6mo agoDoes this use anything from the flash-moe project? https://github.com/Alexintosh/flash-moe https://github.com/Alexintosh/flash-moe
- aegis_camera 6mo agoYes, this is a reference project, the main different is we don't use os swap ( it introduces latency, will add https://github.com/danveloper/flash-moe https://github.com/danveloper/flash-moe to the original reference as well ).
- aegis_camera 6mo agoWe implemented two techniques to run massive 100B+ parameter MoE models natively on the M5 Pro 64GB MacBook Pro: TurboQuant KV compression: We ported the V3 Lloyd-Max codebooks from the TurboQuant paper (Zandieh et al., ICLR 2026) into native C++ and fused dequantization into Metal shaders. This achieves a measured 4.3× KV cache compression at runtime, completely eliminating Python overhead. SSD Expert Streaming: To fit a 122B parameter model (e.g., Qwen3.5-122B MoE) without triggering macOS VM swapping or Watchdog kernel kills, the full ~60 GB weight file remains on NVMe. Only the top-k active expert pages are streamed to the GPU per forward pass at ~9 GB/s. As a result, inference runs with only 2,694 MB of active GPU VRAM on the M5 Pro 64GB, while the OS page cache automatically handles hot-expert reuse. By combining these two approaches, we can comfortably run massive models in memory-constrained environments on Apple Silicon. Also tested QWEN 4B on IPHONE 13 Pro. Code and implementation details: https://github.com/SharpAI/SwiftLM https://github.com/SharpAI/SwiftLM
- altruios 6mo agowhat tokens/s are you getting with a 122B MoE model in this setup? I didn't see any benchmarks in the benchmarks section on the readme.md
- gigatexal 6mo agoyeah this I'd like to see added to teh readme.
- aegis_camera 6mo agoI'll add more details. We just wired up the pipeline on both MAC and IOS.
- aegis_camera 6mo agohttps://www.sharpai.org/benchmark/ https://www.sharpai.org/benchmark/ The MLX part is what we've done with SwiftLM, the local result is still being verified more details are on-going.
- anemll 6mo agoCheck it out, you might be able to speed it up using this https://github.com/Anemll/anemll-flash-mlx https://github.com/Anemll/anemll-flash-mlx https://x.com/anemll/status/2038684375425200360 https://x.com/anemll/status/2038684375425200360
- aegis_camera 6mo agoThanks, pure Swift was the design idea and since I found nothing could be used for my project https://www.sharpai.org https://www.sharpai.org then I created Swift version. Python is too heavy to be delivered with application, user mentioned they want to use MLX, that's why I've been working on it for 1-2 weeks for bug fixing and testing , then suddenly TurboQuant proposed, I had a quick integration. My 64GB M5 Pro is already good for my local security task, now it's able to use M1/M2 Mini w/ 8GB memory.
- Aurornis 6mo agoAlthough I'm interested in both topics (KV compression and attempts to stream MoE models from storage) this is at least the 10th vibecoded project on this topic I've seen today alone across HN, Twitter, and some subreddits I visit. At least this one gave credit to the upstream projects which it used as a reference. The llama.cpp project is also getting a wave of vibecoded PRs that are very clearly being produced by pointing claude at the repo and the original paper and having it produce something. Almost none of these attempts contain information that really matters, like actual benchmark tests with differen KV quantization levels (not just perplexity or KLD).
- _zoltan_ 6mo ago"vibe coded" is NOT the bad thing you think it is. Going from paper to implementation from scratch in half an hour or so is great.
- mjr00 6mo ago> "vibe coded" is NOT the bad thing you think it is. It's not inherently bad in the same way that a first draft of a novel is not inherently bad. But if someone asked me to read their novel and it was a first draft that they themselves had clearly not bothered reading or editing, I'd tell them to fuck off.
- sumeno 6mo agoAt least in the novel example the author had the decency to write what they're asking you to read. These are more like sending someone who didn't ask you a question a LMGTFY link they didn't ask for and expecting them to read all the results. Just a complete lack of awareness and respect for the maintainers
- brokencode 6mo agoThat’s a starting spot, but how about some testing and benchmarks? Where’s the value added if the person just tells Claude to do it and then submits a PR? The maintainers may as well vibe code it themselves if that’s all the work the would-be contributor is going to put into it.
- vessenes 6mo agoI like this idea on expert streaming. I've been poking around fairly thoroughly at the same idea - can we fix a set of experts? when can we fix them? How long is the top-k selection "good" for in terms of number of forward passes? One thing I've turned up in smaller models and I'm sort of winding my way toward verifying in larger ones is that if you train the MoE model from scratch with this kind of knockout / subset of experts baked in, then you get significantly better loss outcomes. In small models, it's actually better than training an MOE without conditioning on a reduced set of experts per pass. Anyway, pretty cool. There's some Pareto-optimal curve based on memory bandwidth, amount of GPU / unified RAM and inference compute times for streaming stuff in.
- aegis_camera 6mo ago[flagged]
- robotswantdata 6mo agoFeels 100% vibe coded in a bad way. Llama.cpp already has KV compression and one of the turbo quant PRs will get merged at some point. If you don’t care about the fancy 3 bit, the q8 KV compression is good enough! Don’t bother with q4 ./build/bin/llama-server -m model.gguf \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ -c 65536 Etc
- aegis_camera 6mo agoOne of my user requested MLX comparison with GGUF, he wanted to run the benchmark, I was thinking about how to get MLX support without bundling the python code together with SharpAI Aegis, a Local or BYOK local security agent https://www.sharpai.org https://www.sharpai.org. Then I had to pick up the Swift and create it. The benchmark shows a benefit of MLX engine, so it's user's choice which engine to use, aegis-ai supports both : )
- xiphias2 6mo agoAnother project without running real benchmarks. It's very easy to generate tokens, it's much harder to solve tasks locally.
- aegis_camera 6mo agoHere is a reference https://www.sharpai.org/benchmark/ https://www.sharpai.org/benchmark/ For specific tasks, local model could achieve workable level.
- simonw 6mo agoI couldn't get the downloadable binary to work, or the binary I compiled myself: ./SwiftLM \ --model mlx-community/Qwen3.5-122B-A10B-4bit \ --stream-experts \ --port 5413 Error: [SwiftLM] Loading model: mlx-community/Qwen3.5-122B-A10B-4bit [SwiftLM] Enabled Async SSD Streaming on directory: e9c67b08899964be5fdd069bb1b4bc8907fe68f5 [SwiftLM] Memory strategy: FULL GPU (69.6GB model, 133.4GB available) [SwiftLM] Download: [===================>] 100% ⠋ (66395.4 MB / 66395.4 MB) | Speed: 0.0 MB/s MLX error: Failed to load the default metallib. library not found library not found library not found library not found at /Users/runner/work/SwiftLM/SwiftLM/LocalPackages/mlx-swift/Source/Cmlx/mlx-c/mlx/c/stream.cpp:115
- deleted 6mo ago[deleted]
- aegis_camera 6mo agogit clone https://github.com/SharpAI/SwiftLM https://github.com/SharpAI/SwiftLM # no --recursive needed cd SwiftLM swift build -c release ### Please let me know if this fix the issue: # Copy metallib next to the binary (one-time step) cp LocalPackages/mlx-swift/Source/Cmlx/mlx/mlx/backend/metal/kernels/default.metallib \ .build/release/
- deleted 6mo ago[deleted]
- simonw 6mo agoClaude Code helped me figure out this recipe (inspired by a similar workaround in the CI scripts): git clone --recursive https://github.com/SharpAI/SwiftLM.git cd SwiftLM swift build -c release # Trick to copy in that missing mlx.metallib file uv run --with mlx-metal python -c " import importlib.metadata, pathlib, shutil d = importlib.metadata.distribution('mlx-metal') metallib = pathlib.Path(d._path).parent / 'mlx/lib/mlx.metallib' shutil.copy(metallib, '.build/release/') print(f'Copied {metallib} -> .build/release/mlx.metallib') # Now start the server (downloads 69GB Qwen model) .build/release/SwiftLM \ --model mlx-community/Qwen3.5-122B-A10B-4bit \ --stream-experts \ --port 5413 But the server crashed when I tried to run a prompt through it: freed pointer was not the last allocation
- gervwyk 6mo agoAnyone else looking at these developments and thinking that local llms are the future. So many advantages above remote, and the hardware is just not there jet, but another leap like apple silicon and the tech is there.. Ofcourse large corps will have fancy proprietary models, but for every day queries and tasks, local feels like a huge, and just slightly out of reach. Am i missing something fundamental?
- cl0ckt0wer 6mo agollm intelligence seems to be proportional to the ram used. All techniques like this will be used by everyone.
- zozbot234 6mo agoYou can almost always use less RAM by making inference slower. Streaming MoE active weights from SSD is an especially effective variety of this, but even with a large dense model, you could run inference on a layer-wise basis (perhaps coalescing only a few layers at a time) if the model on its own is too large for your RAM. You need to store the KV-cache, but that takes only modest space and at least for ordinary transformers (no linear attention tricks) is append-only, which fits well with writing it to SSD (AIUI, this is also how "cached" prompts/conversations work under the hood).
- daft_pink 6mo agoI’ve always believed local is the future. If you consider how your iPhone has a processor that is more powerful than something very large not too long ago.
- aegis_camera 6mo agoI've ran this on an IPHONE 13 pro (6GB) memory, QWEN 3 1.7B runs good. So local will get more intelligent for the task you want it done soon or already.
- daft_pink 6mo agoCan this work on M1, M2, M3, M4?
- aegis_camera 6mo agoYes, I've ran it on IOS, IPHONE 13 pro beside M5 pro, I'll test it on my M2 Mini and M3 Air.
- dalemhurley 6mo ago[dead]