6 ms·
We don’t know what quantization level was used for the weights or the kv cache for you or for parent poster, so this is probably an apples to oranges comparison
by cellularmitosis 27d ago
We don’t know what quantization level was used for the weights or the kv cache for you or for parent poster, so this is probably an apples to oranges comparison.
- spacebacon 27d ago[dead]