4 ms·

Try Ollama or LM Studio. Mixtral and its finetunes work perfectly for me on my RTX3090 with offloading.