8 ms·
The rule of thumb is roughly 44gb, as most models are trained in bf16, and require 16 bits per parameter, so 2 bytes. You need a bit more for activations, so ma
by fnbr 2y ago
The rule of thumb is roughly 44gb, as most models are trained in bf16, and require 16 bits per parameter, so 2 bytes. You need a bit more for activations, so maybe 50GB?
you need enough RAM and HBM (GPU RAM) so it’s a constraint on both.
- sharbloop 2y agoWhich GPU card can I buy to run this model? Can it run on commercial RTX3090 or does it need a custom GPU?
- Havoc 2y ago3090 or 4090 will be able to run quantized 22B models. Though realistically for code completion smaller models will be better due to speed
- TechDebtDevin 2y agoEasy..
- Novosell 2y agoMost GPUs still use GDDR I'm pretty sure, not HBM. Do you mean VRAM?