6 ms·
No problem. Varying context size is a common request I've been getting as well. Personally I'm looking forward to seeing how much we can cram into the ancient K
by eso_logic 2mo ago
No problem. Varying context size is a common request I've been getting as well. Personally I'm looking forward to seeing how much we can cram into the ancient K80's 24GB of VRAM :0
- russianGuy83829 2mo agoThank you, looking forward to it. I just saw this simple patch to enable MTP (potentially 2x performance) on older GPUs (Kepler etc), so maybe it will work for you https://github.com/ggml-org/llama.cpp/pull/25680 https://github.com/ggml-org/llama.cpp/pull/25680 Also, for Qwen, the 4 bit _XL quantization seems to have a good balance of performance to size.