6 ms·
isn't it 2 bytes (fp16) per param. so 7b = 14 GB+some for inference?
by hislaziness 2y ago
isn't it 2 bytes (fp16) per param. so 7b = 14 GB+some for inference?
- ancientworldnow 2y agoThis was trained to be run at FP8 with no quality loss.
- hislaziness 2y agoThe model description on huggingface says - Model size - 12.2B params, Tensor type - BF16. Is the Tensor type different from the training param size?
- fzzzy 2y agoit's very common to run local models in 8 bit int.
- qwertox 2y agoYes, but it's not common for the original model to be 8 bit int. The community can downgrade any model to 8 bit int, but it's always linked to quality loss.