5 ms·
Interesting, I had assumed it'd be too large to fit. What quant and context size are you running?
by fluoridation 7d ago
Interesting, I had assumed it'd be too large to fit. What quant and context size are you running?
- tarruda 7d agoIQ3_XXS (~3.2 BPW). For me this is an option because my Mac studio is only used for serving LLMs, so I can afford to dedicate most of its RAM to this. I can run with 256k context and only uses ~117G, with the remaining (up to 125G which I can allocate to VRAM) being used for prompt caching and context checkpoints. I'm making my own quants, though the Vision-Exp version is outdated and won't work on llama.cpp master branch (I built it before llama added support): - https://huggingface.co/tarruda/DeepSeek-V4-Flash-0731-GGUF https://huggingface.co/tarruda/DeepSeek-V4-Flash-0731-GGUF - https://huggingface.co/tarruda/DeepSeek-V4-Flash-Vision-Exp-GGUF https://huggingface.co/tarruda/DeepSeek-V4-Flash-Vision-Exp-... For the Vision-exp version, I also ran perplexity + KLD against the original MXFP4. Seems quite OK: https://huggingface.co/tarruda/DeepSeek-V4-Flash-Vision-Exp-GGUF/blob/main/logs/perplexity-IQ3_XXS.txt https://huggingface.co/tarruda/DeepSeek-V4-Flash-Vision-Exp-...
- fluoridation 7d agoThanks, I'll give that a try. I basically have the same use case, only on Strix Halo.
- Darnoq 6d agoPls share info if you make it work
- fluoridation 5d agoIt didn't work. DeepSeek failed to load on LM Studio due to a tensor format error. I don't have the time to do deeper troubleshooting. Qwen 3.8 Next did work, but the performance was abysmal.
- tarruda 7d agoDon't use my Vision-Exp GGUF though. As I said I built those GGUFs before llama.cpp supported, and they can't be loaded on current master (require my own branch). I already have new GGUFs but haven't uploaded yet. If you want Vision-Exp, maybe use bartowski or unsloth's GGUFs. Side note: As an alternative to deepseek v4, you might want to give it a shot at qwen 3.8 flash next. I have IQ4_NL GGUFs that can be loaded fully into 128G, or Q5_K GGUFs that can offload the PLE to disk (use --load-mode none --lazy-mode on for that): https://huggingface.co/tarruda/Qwen3.8-Flash-Next-GGUF https://huggingface.co/tarruda/Qwen3.8-Flash-Next-GGUF. llama.cpp master is still somewhat bad in Qwen 3.8 next performance, but I was able to achieve 40tps tg and 600 tps pp on my private branch.
- kamranjon 7d agoHey there! I do the same but I use dwarfstar at a 2-bit quant: https://github.com/antirez/ds4 https://github.com/antirez/ds4 I'm curious if you've tried dwarfstar and decided to move to llama.cpp and 3 bit quants or what made you go that route instead? I've been using ds4 for months now and it's already got support for the new vision model, haven't tried it yet, still on 0731 but it's been very solid for me.
- tarruda 7d agoI tried dwarfstar when llama.cpp DSV4 support was still very weak, and while it worked, I didn't see anything that would make me want to stick with it vs llama.cpp. llama.cpp is simply better with its awesome built-in webui, router and server APIs and certainly support much more models and quantizations than dwarfstar. Since then, I started maintaining my own vibe coded dsv4 branch with metal optimizations, so I actually get much better metal performance on my llama.cpp branch than on dwarfstar (plus all the extra llama.cpp features). Here it is in case you want to give it a shot: https://github.com/tarruda/llama.cpp/tree/qwen4exp-dsv4-optimizations https://github.com/tarruda/llama.cpp/tree/qwen4exp-dsv4-opti...