15 ms·
Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s
I built slotstream, a way to run Qwen3.8-Flash-Next 4-bit on a low-memory mac starting from 16GB, a 125B parameter model that would need 100GB+ memory/RAM, thanks to expert-offloading/ssd-streaming. Easy to install/update, and mac-native using MLX and Swift.
It ships with auto-mode, which makes a good tradeoff between memory usage and speed. I'll be implementing and porting the MTP module for speculative decoding next
- vancekai 14d ago[dead]
- securecloudgrou 13d agoWould love to see metrics of model performance, comparison with oMLX/OLLAMA/others. Any tooling for local optimization on hardware.
- securecloudgrou 13d agoTo be more clear, I see you have slotstream doctor --sim-ram N, but extended tooling and optimization for exact local hardware would add value (MTPLX has a nice interface for example).
- siris9476 15d ago32GB dedicated to an N-gram table instead of a draft model is an unusual choice for speculative decoding — what made it win over the more common draft-model approach here?
- carloslfu 13d agoIt wasn't either/or, the N-gram table is part of Qwen itself and stays on disk. I’ve now added its 1.5GB MTP draft head too, it gets 86% acceptance and about 1.24× faster decoding on my 48GB Mac.
- siris9476 13d agoGot it, thanks for clearing that up. 86% acceptance is solid — does it stay flat over longer generations, or drift with context length?
- amelius 15d agoHow usable is 12 tok/s?
- pornel 15d agoUnpleasant for interactive agentic work. Still useful to leave it to do some work in the background.
- c0rruptbytes 15d agoso many inference project, omlx already supports all of this and has a 1000 people trying to optimize it constantly
- jmward01 15d agoNot a mac/UMA discussion point, but is it time to add additional, installable, DDR5 to GPUs? I can see this as a win/loose. PCIe 5x16 is close to maxing out the bandwidth available from high end dual channel DDR5 now, but not quite. I'm not a hardware person but I suspect putting it on the card could lead to significant performance improvements over using system ram so allowing systems like this, where MOE weights are shed, to get even higher performance than just adding that DDR5 to the system. Bigger models become closer to reality and it provides more of a pathway for developing technologies that take advantage of it. Of course the loose side is that you just put a lot of specialized ram on a card instead of into the system where it could be used for other things. I could see a place for a 16GB card with 64GB(or more) of DDR5 especially if we start seeing MOE and similar technologies really start being designed for this concept.
- MayeulC 15d agoProbably not with DIMM modules, as longer traces mean higher latency (speed of light is ~30 cm in 1ns). GDDR typically uses larger buses (more wires) for higher bandwidth, even more so for HBM, so DIMM would be hard. Maybe CAMM would be up to the task? It certainly seems feasible from an engineering perspective (though it does make cooling harder), at least for mid-range, not H100-class HW, but it prevents market segmentation, so EOMs may not be too interested (as long as no competitor does it).
- workletter_co 15d ago[flagged]
- sriniwasx 15d ago[dead]
- jwr 15d agoFor what it's worth, I've spent some time with Claude to develop a local runner for `llama.cpp`. I run Qwen3.6-35B-A3B-MTP (fast!) and Qwen3.8-27B (20 tokens/s). This was definitely worth the effort. Benchmarking and testing various approaches and various options really paid off. For example, one thing that surprised me was that MTP made things slower, not faster for Qwen3.8-27B. I use a 64GB MacBook Pro (M4 Max).
- nixon_why69 15d agoI find mtp=3 does well with that model, only at 4 it becomes unprofitable. Check your quants, its worth having the mtp layer be a bigger quant if it leads to 2x throughput from more accepted tokens.
- ch_sm 15d agoI’m not an expert, but my understanding is that MTPs are smaller LLMs fine-tuned to "mimic" / predict a specific model’s response. It’s possible that the MTP you’re using isn’t trained well enough on Qwen 3.8. What accept rate are you getting?
- nixon_why69 14d agoFor qwen, it's an additional transformer layer at the very back, it ships as part of the model.
- jwr 11d agoAcceptance rate is good, but MTP doesn't help in my case because of my machine's memory bandwidth constraints (M4 Max). Turn out it's better to turn MTP off.
- baristaGeek 14d agoI just used it, went through the whole installation (took like 1 hour approx). Long but straightforward. If I put my computer to sleep will it continue? I started the server (very curiously I was running oLlama in the same prot slotserve uses by default, instead of switching it which I know you can do, I just ditched oLlama, perhaps an insight for you) and built a small html hello world served via Python. The thing pointed me to the localhost link, nice! As an early user, my advice is to focus on efficiency. The efficiency of the installation but more importantly, the efficiency of running the thing. 8.1GB per slotserve process is a lot! Is that in your control? Also, I've seen an interest of certain kinds of programmers for open-weight models. "We all know agree that LLMs for coding are very useful but we're giving money to a small set of big, evil corporations. They're Trump donors. I heard it's bad for the environment because it uses water". If it's local and open-weight, this could be marketed this way I think. Finally, what's the actual, real use case for slotserve?
- carloslfu 14d agothanks! > ditched oLlama" yeah! this is interesting. > 8.1GB per slotserve process is a lot! Is that in your control? yes, it is hard, but I agree the smaller the better. I'll work on that > If it's local and open-weight, this could be marketed this way I think. I like this! > what's the actual, real use case for slotserve? I'm working rn on an app on top of it that closes the loop and is a fully local AI app, an experiment. I'll publish it as soon as it is usable! > built a small html hello world served via Python What did you use as a harness here?
- baristaGeek 14d agoFor the harness... just the shell. No client library. Does that answer your question?
- carloslfu 14d agothanks! in part, I was wondering how you got the code into files. I guess you copy pasted it inside a file, am I right?
- baristaGeek 14d agoAlso I don't know if people like this... but could this be wrapped around a nice desktop app?
- carloslfu 14d agogreat idea!! a native app would be awesome
- NikhilChowdaryG 14d ago[flagged]
- nerdsnipe 14d agoBrilliant! Have you had any success integrating it into a MacOS Swift app. I'd love to see it in action before I consider adding it to a future build. My biggest issue is getting these opensource models to use tools well enough for production.
- carloslfu 14d agoThis is next! in the works rn.
- jedbrooke 14d agoI tried this in my mac mini m2 16GB, unfortunately I have to use an usb disk for the model weights, and I’m getting 0.5 tok/s. Still, being able to run (heh maybe crawl is more accurate) a 100B model on this computer AT ALL is pretty cool. I see disk maxing out at 400 MB/s, this disk should be able to hit 1GB/s (it hits that eg when verifying the check sum of the weights), so there might be some optimization to be done there (I’m guessing it’s because the weights access is not pure sequential reads but involves some randomness depending on which expert)
- AmazingTurtle 15d agoThere are already a handful of repos doing essentially exactly this: `mlx-moe-offload`, `streamlx`, `mlx-moe`, `mlx-flash`, and `deepseek-v4-flash-mlx` - i.e. keep the resident parts of an MoE in unified memory and page/stream routed experts from SSD on Apple Silicon. At this point I'd much rather see people collaborate on one of these implementations, benchmark against them, or upstream the useful bits into MLX/MLX-LM instead of producing yet another near-identical repo. The local-LLM ecosystem really does not need every implementation idea rediscovered five times and wrapped in a new README. AI-assisted coding makes producing a new repo cheap; maintaining, benchmarking, and integrating one is the actually valuable part.
- kzrdude 15d agoAnd there are `Mference` and `SwiftLM` too, I think they are doing the same use case.
- api 15d ago> every implementation idea rediscovered five times and wrapped in a new README That's open source since forever, unfortunately.
- docheinestages 15d agoIt's what happens when you don't do market research.
- ErenayDev 15d agohow much energy does it consume?
- carloslfu 15d agoGood one! I haven't measured this. I'll include it!
- karmakaze 15d agoIt seems we could use a new kind of memory that streams the weight data in, like GDDR in reverse.
- 0x457 15d agoHigh Bandwidth Flash? https://www.sandisk.com/company/newsroom/blogs/2025/scaling-beyond-the-wall-inside-sandisks-high-bandwidth-flash-for-ai https://www.sandisk.com/company/newsroom/blogs/2025/scaling-...
- carloslfu 15d agoyes! I guess future hardware designs will have something like that!
- drcongo 15d ago"Disk is the gate that bites first" AI;DR
- embedding-shape 15d ago> Hugging Face is the bottleneck, not your link. README could clearly make use of a cleanup, seems to be more like a session log dump now than a good introduction to the project for a new user. Maybe try something like "Remove anything from the README.md that wouldn't be helpful to someone who sees this project with zero context, for the first time. Rewrite all paragraphs and sections to be concise and remove all fluff, leave only important details new users must know before using the project".
- ricardobeat 15d ago> Run Qwen3.8-Flash-Next on a Mac that can't hold it This is the first line of the README. I can't believe people are becoming ok with this, and I'm 100% on the AI train.
- carloslfu 15d agothanks! I'll do!
- xlayn 15d ago[flagged]
- embedding-shape 15d ago> if someone completely new comes and read the current page... isn't that piece of information something they want to know? Again, the README is (was?) written in way that it was clear it was a conversation between an agent and a human, with literal replies in the README. I'm in no way whining the way you are or demonstrate, and I even give OP helpful advice to fix it. What more can you want from me?
- Eufrat 15d agoI hate this AI style writing because since it doesn’t really understand flow, it’s being inserted in irrelevant places and it is extremely irritating to read.
- 15d ago
- bewareofscams 15d ago[dead]
- bewareofscams 15d ago[dead]
- whartung 15d agoI'm hoping to see progress in this space. Folks talking about how 32G is not enough for local use, but then there's been work like this to empower it. My hope is that the new 32G M6 will be "useful" locally, possibly because of work like this.
- carloslfu 15d agoyes! I'm bullish on this. there is a lot of work to do. I've been experimenting with pruning, distillation, and retraining too. I'm sure your 32gb m6 will run a badass local model!
- tyre 15d agoYes, but also 12 tok/s versus Claude is so far from comparable. I know that it’s not exactly 1:1, but it’s a long way from an easy trade-off, especially considering hardware prices for high levels of RAM.
- trollbridge 15d ago32GB is simply too tight; you need 8 minimum for the OS and you need about 4-8 more for the LLM you’re visiting and kv cache.
- prometheus1992 15d agoIt's hard to believe 16GB unified memory will give you 5 tok/sec unless you are ignoring the thermal warnings. I am running Qwen3.6-35B-A3B on my 16GB M3 and get 7-8 tokens/sec with all the optimizations while keeping the peak memory and thermal warnings at check. https://github.com/deepanwadhwa/samosa-chat https://github.com/deepanwadhwa/samosa-chat
- monster_truck 15d agoThe laptops definitely can't hang but the minis don't really care. I threw mine down in the basement just to put the heat somewhere else, can tell when the dehumidifer next to it is on because it's a few C lower but that has no impact on performance. I don't think it's ever seen anything north of 70
- carloslfu 15d agointeresting! Yes, thermal is important. Pretty cool project man! Starred and checking it out!
- Balooga 15d agoNow I'm feeling pretty good about getting 10-11 tokens/sec running Qwopus 3.6-35B-A3B Q6_K on an old Mac Pro 2013 (trashcan) with 128GB RAM (DDR3), 12 core Xeon, dual D700s. Arch Linux and llama.cpp.
- prometheus1992 15d agohaha, good for you.
- trollbridge 15d agoAnything smaller than a 16” runs into serious thermal problems; even an identically equipped 14” just can’t dissipate enough heat.
- aislopnogo 15d ago[dead]
- atif089 15d agoAs someone who is just looking at the theoretical benchmarks of each of these models I'm curious if anyone could share what are the problems (maybe around code) that flash-next was able to solve which 27b was not able to
- carloslfu 15d agoThis is the best I could find: https://huggingface.co/Qwen/Qwen3.8-Flash-Next?utm_source=chatgpt.com#language https://huggingface.co/Qwen/Qwen3.8-Flash-Next?utm_source=ch... About the specifics, I have only anecdotal evidence, but I guess this info can be found somewhere
- red_hare 15d agoFor a local non-coding agent, instruction following and tool use are the most important gains
- jonplackett 15d agoIs this going to destroy my SSD?
- Gigachad 15d agoA particularly worrying situation considering a dead SSD will render your macbook usable for parts only.
- cromka 15d agoBy reading it?
- carloslfu 15d agoI don't know actually. I'll check haha. My best guess is it isn't.
- carloslfu 15d agoI hope not! this is a new macbook lol!
- mrob 15d agoReading causes insignificant wear ("read disturb") that likely isn't a problem, but I don't think it's possible to issue pure reads to modern SSDs. The NVMe spec mandates tracking the amount of data read, and this has to be written to the drive. I'd hope the firmware buffers this and writes it at low frequency, but on the other hand, I doubt the firmware was tested in extreme random-read regimes. Unexpected failures from excessive statistics recording could be possible.
- egorfine 15d agono it's reading, not writing
- ElectricalUnion 15d agoUsing macos on low memory regimes will make it use disk-based swap. For example, a Macbook Neo (so in theory, something with around 4GiB of free RAM lying around) might eat around 900GB of writes a day while not doing much at all, because it's basically on low on RAM and swapping all the time.
- nikanj 15d agoI swear the models are named by the beatbox aliens from the post office in MiB
- mulemisterX 15d agoI have a 48GB M5. I don't need to run larger models. I want more context. I've managed to set the context window at 71,680 using Qwen3.8-27B-oQ4e-fp16-mtp. But I want more. Is anybody, with similar specs, able to set their context window higher?
- pram 15d agoYou should try Glimmer MTP. Qwen3.8 27B seems to have weird memory and caching issues on oMLX
- carloslfu 15d ago[dead]
- hadlock 15d agoWe are running 35b-A3b with 264k context (the model's default max) using vllm and the "frog" jinja templates: https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates and had good luck. We are mostly running agentic workloads though, rather than coding. 27b has a slightly higher agentic job completion rate (95% vs 92%) but the 3% trade off is worth it because the A3B is sooooo much faster, and we reprocess the other jobs with a different model. Don't sleep on the froggeric templates. Qwen: Looking at you for a new ~35B MoE! Please and thank you
- kamranjon 15d agoI am running 3.8 27b at q6 quant with 160k context on a 32gb video card (arc b70 pro) - I quantized the kv cache at q8 - that is the only trick really - works great.
- jonplackett 13d agoHow are you finding it when it gets that high anyway? I’ve got a 64gb Mac so the context _can_ go higher. But I find around 70-80k it goes a bit weird and starts arguing with itself and doesn’t seem to know who it is vs me anymore.
- ig0r0 15d agoyes, with qwen3.8-27b-4bit run via rapid-mlx i can get to about 200k
- GCU_BlessYourLi 15d ago[dead]
- jacquesm 15d agoI love these efforts to get proper models running on lower cost hardware and I think this is where the next real breakthrough will come from. The more efficient this sort of thing can be done the bigger the chance to democratize this tech, 'good enough' is what you need and as long 'top of the line' gives a competitive edge even if it is at a cost there is a substantial risk of the door closing on general computing at some point in the near future. Keep in mind that there is no guarantee that the pendulum has to swing back, it can swing one way and get stuck, and then you're going to have to beg for crumbs from the haves.
- softwarewright 10d agoI am trying to make it easier to use LLMs on older, cheaper, smaller GPUs. I'm taking a similar approach (move MoE expert weights to disk, avoid wasting VRAM on these). My goal is also to run models that do not fit. My work also suffers from AI documentation issues. Where my approach differs is that instead of running an LLM that doesn't fit slowly, run many agents in parallel sharing the streams of MoE experts weights, to increase throughput. I envision a team of AI agents sharing a pretty-good-at-coding LLM that does not fit to collaborate on a set of related features, being developed in parallel. My work is showing promising results (if you can get past the way the AI tries to describe what I am doing). https://sw-ml-study.github.io/emufpga/index.html https://sw-ml-study.github.io/emufpga/index.html I am doing this work initially on a 6-Xeon-cores Linux workstation with an RTX5060-16G to run MoE models larger than that. Then I will be moving this to a server with a lot more cores (Dual 32-cores) and a mix of SAS HD and SSD drives, using older GPUs. Ultimately, I hope to build some FPGA/MCU "accelerators" that process the expert weights on systems with not enough CPU cores to offload the experts. If I can enable large capable models to run on older hardware, keeping the limited GPU VRAM for context and things that must be in VRAM, I can get useful work out of my old refurbished systems without paying today's RAM and VRAM/GPU prices.
- cosmic_cheese 15d agoI think there's a very good chance that history will rhyme a bit. DOS/Windows and PC clones were by no means the best available, but they were cheap, ubiquitous, and versatile compared to alternatives that were either much better at one task but more expensive or better at everything but wildly expensive. They were "good enough" and represented a solid improvement over what many existing computer users had as well as a good entry point for new users. As such they spread like wildfire and became the standard while the expensive alternatives either became hardcore niche or vanished.
- kethinov 15d agoNext help us normies run GLM 5.3 on our potato computers. Wouldn't that be nice!
- kzrdude 15d agoColibri did that first for GLM-5.2 https://github.com/JustVugg/colibri https://github.com/JustVugg/colibri