7 ms·

Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model. Today I was trying in fact to see, what's the smallest Qwen3.8-27B th

by xlayn 29d ago

Hey Unsloth, your gguf are the first ones I look for when I want to download a gguf model. Today I was trying in fact to see, what's the smallest Qwen3.8-27B that I could run and get good results, say restricting it to 16GB of ram.. so I went, pick up the Qwen3.8-27B-UD-IQ2_XXS.gguf and them BAM, error on MTP... now I understand why after reading your announcement. Beyond the space saving, why removing the MTP? improves speed exactly for the group that could benefit from it.

mike-the-brain 29d ago
you can still have it, no? > We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed
- xlayn 29d ago
  my bad, you are totally right, thanks!
gruturo 28d ago
The reason for running those insanely low quants is to fit in extremely limited memory budgets. The first thing you sacrifice is speed, then context and accuracy (up to you in which order). IQ2_XXS and below is desperate/proof of concept territory. If you have a spare half gig for the MTP drafter, run a larger quant instead, it will be less incoherent, and damn the speed, it won't be garbage at least. Only around Q4 I'd allocate the comparative luxury of more memory for a speed increase. At least on a dense model. MTP makes a lot more sense (but helps statistically a bit less) on an MoE. Qwaiting for that 3.8-35B-A3B
danielhanchen 28d ago
Hey we did not remove the MTP for sizes above 8GiB - but yes for small GGUFs under 8 ish GiB, we removed the MTP module (IQ2_XXS and lower), because it's 500MiB to 750MiB in size, and on small 8 GiB machines, even 500MiB is needed. As someone in the comments said we made a separate Q4_0 MTP if that's helpful so you can use that. But I would suggest using UD-IQ3_XXS for 10.9GB for 16GB machines or Q2_K_XL
- xlayn 28d ago
  Daniel, question I got the Qwen3.8-27B-UD-Q2_K_XL.gguf from https://huggingface.co/unsloth/Qwen3.8-27B-GGUF?show_file_info=Qwen3.8-27B-UD-Q2_K_XL.gguf https://huggingface.co/unsloth/Qwen3.8-27B-GGUF?show_file_in... and continue with my testing, but the model quickly felt into a loop of asking the same thing over and over again, I have seen the MOE do that but not the dense ones. And I had similar experiences when Qwen3.8-27B unsloth images just came out with the full Q8_K_XL, I'm using an AMD setup which has modifications to save to disk the kv, but your (assuming you are part of the unsloth team) for some reason have been giving me similar issues. I tried https://huggingface.co/mradermacher/Qwen3.8-27B-Uncensored-GGUF https://huggingface.co/mradermacher/Qwen3.8-27B-Uncensored-G... the 8 bit, 6 and 2 bit... the 2 bit almost use the complete KV doing it's thing and didn't loop itself. It can be something in my setup, there is a very high chance of that, but the previous 3.6 images from qwen, the 27B, the 31A3 and 122 they are all unsloth and did work on my setup without issues... Again could be my setup... let me know if there is any data I can supply to you to debug if needed.
  - kamranjon 28d ago
    Are you using the recommended settings for temperature and such? https://unsloth.ai/docs/models/qwen3.8#recommended-settings https://unsloth.ai/docs/models/qwen3.8#recommended-settings Often times I run into issues like this it’s because I am using settings for a different model or just forget to set them up.
- idonotknowwhy 28d ago
  >But I would suggest using UD-IQ3_XXS for 10.9GB for 16GB machines or Q2_K_XL For those of us with a 16GB GPU, how do they compare with ExllamaV4 at 4-bit (4.0bpw)? It looks like that fits in 12.5GB of VRAM since embedding are left in DRAM, Unsloth Studio and other llama.cpp derivatives have to load these weights in VRAM for tied embedding models like Qwen3.8. ExllamaV3 4.0bpw fits in 12.5G of VRAM and beats IQ4_XS according to the measurements here: [turboderp/Qwen3.8-27B-exl3](https://huggingface.co/turboderp/Qwen3.8-27B-exl3 https://huggingface.co/turboderp/Qwen3.8-27B-exl3) But those were compared against UD2.0 I guess. Also plans to support these (SOTA) quants in Unsloth Studio?
walrus01 28d ago
Q2 quantization is basically giving a capable model a lobotomy. It will not accurately represent how smart or capable something like qwen 3.8 27B in Q8 will be.
- zenoprax 28d ago
  Sure, but this is true for all lossy compression (audio, images, etc.) Given 16GB of VRAM, what will give me the best experience in OpenCode? Currently using Qwen3.8_Q_3
  - walrus01 28d ago
    probably the best experience would be deepseek v4 flash 0731 (it takes about 170GB RAM on the server side for the full thing and RAM reserved for 1M context) via opencode's $10 a month plan until you use that up, it's either Q8 or full precision. Assuming you're ok with doing things with external inference.
    - kennywinker 28d ago
      Why would they have listed how much vram they had if they were looking to rent gpu time on someone else's machine?
      - walrus01 28d ago
        A casual review of my comment history would show that I've been nothing but the biggest proponent of running models locally, and I do so myself a great deal. But one also has to be realistic about the capabilities of what you can do in a 16GB GPU these days. I already said an extra small Q2 quantization was effectively lobotomized so I didn't want to repeat myself. This person has basically run into the limit of state of the art for even a modestly sized local model (this isn't deepseek v4 flash 0731 Q8 which I am running myself locally on a great deal more hardware), this is a 27B dense, but they're just not going to have a good time if they expect good quality results out of a Q2. The choices are either upgrade hardware or pay for external inference.

by xlayn 29d ago

mike-the-brain 29d ago

you can still have it, no? > We also removed the MTP module from smaller quants under UD-Q2_K_XL (8.37GB and lower) to converse around 500MB of disk space - you can use the Q4_0 MTP separate module if needed

xlayn 29d ago
my bad, you are totally right, thanks!

gruturo 28d ago

The reason for running those insanely low quants is to fit in extremely limited memory budgets. The first thing you sacrifice is speed, then context and accuracy (up to you in which order). IQ2_XXS and below is desperate/proof of concept territory. If you have a spare half gig for the MTP drafter, run a larger quant instead, it will be less incoherent, and damn the speed, it won't be garbage at least. Only around Q4 I'd allocate the comparative luxury of more memory for a speed increase. At least on a dense model. MTP makes a lot more sense (but helps statistically a bit less) on an MoE. Qwaiting for that 3.8-35B-A3B

danielhanchen 28d ago

Hey we did not remove the MTP for sizes above 8GiB - but yes for small GGUFs under 8 ish GiB, we removed the MTP module (IQ2_XXS and lower), because it's 500MiB to 750MiB in size, and on small 8 GiB machines, even 500MiB is needed. As someone in the comments said we made a separate Q4_0 MTP if that's helpful so you can use that. But I would suggest using UD-IQ3_XXS for 10.9GB for 16GB machines or Q2_K_XL

xlayn 28d ago
Daniel, question I got the Qwen3.8-27B-UD-Q2_K_XL.gguf from https://huggingface.co/unsloth/Qwen3.8-27B-GGUF?show_file_info=Qwen3.8-27B-UD-Q2_K_XL.gguf https://huggingface.co/unsloth/Qwen3.8-27B-GGUF?show_file_in... and continue with my testing, but the model quickly felt into a loop of asking the same thing over and over again, I have seen the MOE do that but not the dense ones. And I had similar experiences when Qwen3.8-27B unsloth images just came out with the full Q8_K_XL, I'm using an AMD setup which has modifications to save to disk the kv, but your (assuming you are part of the unsloth team) for some reason have been giving me similar issues. I tried https://huggingface.co/mradermacher/Qwen3.8-27B-Uncensored-GGUF https://huggingface.co/mradermacher/Qwen3.8-27B-Uncensored-G... the 8 bit, 6 and 2 bit... the 2 bit almost use the complete KV doing it's thing and didn't loop itself. It can be something in my setup, there is a very high chance of that, but the previous 3.6 images from qwen, the 27B, the 31A3 and 122 they are all unsloth and did work on my setup without issues... Again could be my setup... let me know if there is any data I can supply to you to debug if needed.
- kamranjon 28d ago
  Are you using the recommended settings for temperature and such? https://unsloth.ai/docs/models/qwen3.8#recommended-settings https://unsloth.ai/docs/models/qwen3.8#recommended-settings Often times I run into issues like this it’s because I am using settings for a different model or just forget to set them up.
idonotknowwhy 28d ago
>But I would suggest using UD-IQ3_XXS for 10.9GB for 16GB machines or Q2_K_XL For those of us with a 16GB GPU, how do they compare with ExllamaV4 at 4-bit (4.0bpw)? It looks like that fits in 12.5GB of VRAM since embedding are left in DRAM, Unsloth Studio and other llama.cpp derivatives have to load these weights in VRAM for tied embedding models like Qwen3.8. ExllamaV3 4.0bpw fits in 12.5G of VRAM and beats IQ4_XS according to the measurements here: [turboderp/Qwen3.8-27B-exl3](https://huggingface.co/turboderp/Qwen3.8-27B-exl3 https://huggingface.co/turboderp/Qwen3.8-27B-exl3) But those were compared against UD2.0 I guess. Also plans to support these (SOTA) quants in Unsloth Studio?

walrus01 28d ago

Q2 quantization is basically giving a capable model a lobotomy. It will not accurately represent how smart or capable something like qwen 3.8 27B in Q8 will be.

zenoprax 28d ago
Sure, but this is true for all lossy compression (audio, images, etc.) Given 16GB of VRAM, what will give me the best experience in OpenCode? Currently using Qwen3.8_Q_3
- walrus01 28d ago
  probably the best experience would be deepseek v4 flash 0731 (it takes about 170GB RAM on the server side for the full thing and RAM reserved for 1M context) via opencode's $10 a month plan until you use that up, it's either Q8 or full precision. Assuming you're ok with doing things with external inference.
  - kennywinker 28d ago
    Why would they have listed how much vram they had if they were looking to rent gpu time on someone else's machine?
    - walrus01 28d ago
      A casual review of my comment history would show that I've been nothing but the biggest proponent of running models locally, and I do so myself a great deal. But one also has to be realistic about the capabilities of what you can do in a 16GB GPU these days. I already said an extra small Q2 quantization was effectively lobotomized so I didn't want to repeat myself. This person has basically run into the limit of state of the art for even a modestly sized local model (this isn't deepseek v4 flash 0731 Q8 which I am running myself locally on a great deal more hardware), this is a 27B dense, but they're just not going to have a good time if they expect good quality results out of a Q2. The choices are either upgrade hardware or pay for external inference.