6 ms·
In my experience if you're coding or doing something that requires precision, quantizing the kv cache is definitely not worth it. If you're just chatting or do
by flux3125 5mo ago
In my experience if you're coding or doing something that requires precision, quantizing the kv cache is definitely not worth it.
If you're just chatting or doing less precise things it's 1000% worth it going down to Q8 or sometimes even Q4