5 ms·
It can be run on an A40 or A6000, as well as the largest A100s. But other than that, no.
by stellaathena 5y ago
It can be run on an A40 or A6000, as well as the largest A100s. But other than that, no.
- djoldman 5y agoHow much VRAM does it use during inference?
- stellaathena 5y ago~40 GB with standard optimization. I suspect you can shrink it down more with some work, but it would require significant innovation to cram it into the next largest common chip size (24 GB, unless I’m misremembering)
- komuher 5y agoIs 40GB already on float16?
- stellaathena 5y agoYes
- bm-rf 5y agoYou could use Microsoft's DeepSpeed to run the model for inference on multiple GPUS, see https://www.deepspeed.ai/tutorials/inference-tutorial/ https://www.deepspeed.ai/tutorials/inference-tutorial/