6 ms·
Yeah I am using the default training script with int8 quantisation. It uses peft with lora but this still requires 26gb
by fbodz 3y ago
Yeah I am using the default training script with int8 quantisation. It uses peft with lora but this still requires 26gb
- freeqaz 3y agoIs it possible to offload some layers to CPU and still train in a reasonable amount of time?
- generalizations 3y agoThere’s also that pruning tool that was on hn in the last couple weeks. It seemed to work really well on the larger models, and could reduce size by 30-50%
- int_19h 3y agoI'm not sure about this model specifically, but training with 4-bit quantization has been a thing with LLaMA for a while now, although the setup involves manual hacks of various libraries.