5 ms·
the model says 8x7B model, so its a 56B model. what is the GPU memory requirements to run this model for a 512 context size? are there any feasible quantization
by matrix2596 3y ago
the model says 8x7B model, so its a 56B model. what is the GPU memory requirements to run this model for a 512 context size? are there any feasible quantization models of this available? I want to know if my 16GB VRAM GPU can run this model?
Thanks
- Jackson__ 3y agoAccording to https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF https://huggingface.co/TheBloke/Mixtral-8x7B-v0.1-GGUF : 18.14GB in 2bit, which is still too high for your GPU, and most likely borders on unusable in terms of quality. You could probably split it between CPU and GPU, if you don't mind the slowdown.