8 ms·
I got a 48GB Ram MacBook, somehow I cannot even run a 20b model, I was suprised that you get 35b model locally.
by timmit 3mo ago
I got a 48GB Ram MacBook, somehow I cannot even run a 20b model, I was suprised that you get 35b model locally.
- klardotsh 3mo ago4-5 bit quants would probably fit pretty well on your rig. Check HuggingFace for Qwen3.6-35B-A3B-MTP-GGUF [1]. They've also got a cool UI thing these days to help indicate which quants of a model will run on your hardware. Full octane isn't gonna fit on much of anything south of a 128GB machine once adding KV cache. [1]: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF https://huggingface.co/unsloth/Qwen3.6-35B-A3B-MTP-GGUF