5 ms·
> Running a large model locally comes down to one thing: how much RAM it actually needs in memory. Not completely true. It's memory AND memory bandwidth. You c
by c16 14d ago
> Running a large model locally comes down to one thing: how much RAM it actually needs in memory.
Not completely true. It's memory AND memory bandwidth. You can have 1tb of memory but if you have awful memory-bandwidth you'll also have slow tok/s. A3B helps with this, but so does MTP.
From my experience, you'd be better off running the dense 27b-mlx with MTP than the 3.6 version with A3B. You say your model is ~20GB of ram, but the 3.8:27b-mlx is 18GB and gets me very reasonable tok/s, and greater speed if you disable thinking when not required.
- Kayou 14d agoThe dense 27b Qwen on M4 Pro has a prompt processing speed of around 125tok/s which makes it ok to ask a quick question but impossible to use in an agent, as processing the first prompt of the agent with the tools and instruction can easily be 10 000 tokens In this case the 35b a3b makes sense as it has a PP speed of around 800tok/s
- c16 14d agoTrue. This then boils down to a quality vs speed decision. the 3.8 27b is far better than 3.6 A3B from my experience. I'm happy taking the speed hit, given local models aren't as intelligent as frontier models. Anything that can get me closer to my CC experience both in reasonable speed and intelligence is worth it. With that said CC can also be slow at times, so it's locally the difference in experience is not always noticeable.
- phazonoverload 14d agoRunning it depends on RAM, which is what I wrote, bandwidth is important for speed. I chose my words carefully, but you are absolutely right.
- phazonoverload 14d agoComing back to this a few hours later, I've decided to add a section to explain this to the blog post. Thank you for flagging it.