5 ms·
Any idea if there is a way to run on 256gb ram + 16gb vram with usable performance, even if barely?
by jdright 1y ago
Any idea if there is a way to run on 256gb ram + 16gb vram with usable performance, even if barely?
- danielhanchen 1y agoYes! 3bit maybe 4bit can also fit! llama.cpp has MoE offloading so your GPU holds the active experts and non MoE layers, thus you only need 16GB to 24GB of VRAM! I wrote about how to do in this section: https://docs.unsloth.ai/basics/qwen3-coder#improving-generation-speed https://docs.unsloth.ai/basics/qwen3-coder#improving-generat...
- jdright 1y agoawesome documentation, I'll try this. thank you!