20 ms·
I get the right answer on the 8B model too. It could be the quantized version failing?
by cbo100 2y ago
I get the right answer on the 8B model too.
It could be the quantized version failing?
- ein0p 2y agoMy models are both 4 bit. But yeah, that could be - small models are much worse at tolerating quantization. That's why people use LoRA to recover the accuracy somewhat even if they don't need domain adaptation.