5 ms·
M1 MacBook Pro is sufficient for a Qwen model.
by deathtrader666 2y ago
M1 MacBook Pro is sufficient for a Qwen model.
- indigodaddy 2y agowith how much ram though?
- int_19h 2y agoFor Qwen-32B, 32Gb should be quite sufficient to run it at 4-bit quantization with full context.
- indigodaddy 2y agoThanks, I need to figure out something to run on M4 Mac Mini standard 16G
- int_19h 2y agoYou should be looking at 7-8b sized models, then. https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct is considered pretty strong for its size. That said, you shouldn't expect more than glorified autocompletion at that point.
- thot_experiment 2y agoI'm not familiar with the practical memory requirements on Macs but I suspect that with 16gb of integrated ram you won't have issues running 14B models even at q6_k, certainly would be fine at q4 and it's definitely going to be capable of writing code based on instruction as well as minor refactoring, generating docstrings etc.
- int_19h 2y agoThe model itself will fit just fine, of course, but you'll also want a large context for coding. And then since it's integrated RAM, it's also used by everything else running on your system - like, say, your IDE, your compiler etc, which are all fairly memory hungry. Also keep in mind that, even though it's "unified memory", the OS enforces a certain quota for the GPU. If I remember correctly, it's something like 2/3 of the overall RAM.