9 ms·

Any suggestions on what GPU to use to train large models?

blackbear_ 5y ago
Totally depends on your budget. The DGX A100 [1] is quite good if you have a fat wallet [1] https://www.nvidia.com/en-us/data-center/dgx-a100/ https://www.nvidia.com/en-us/data-center/dgx-a100/
atty 5y ago
Really depends on what you mean by large. If you mean truly large, you will need a cluster to train it in any reasonable amount of time. You’d probably want to look at servers built on the HGX platform (8 A100s per server). We use servers leased in bulk from traditional server providers (think Dell, HP, etc). If you mean more like “as large as personally affordable”, then you’d probably want to look at something like the RTX 3090, if you can get lucky and find it at MSRP, it has 24 gigs of memory. Nvidia also has their workstation cards with up to 48 gigs if I remember correctly, but if I were buying cards for myself, I would wait until I could get two 3090s somewhere close to MSRP, instead of paying the markup on the workstation cards (unless you want to have more than 2 in a workstation, in which case you’d need to go for those)
lvl100 5y ago
2 x 3090FE is the best bang for your buck.
- cinntaile 5y ago
  Do you need watercooling to keep them from running too hot?
  - maxwells-daemon 5y ago
    I use 2x3090 to train large language models, and mine don't thermal-throttle with air cooling even though they're right next to each other. Eth mining does generate too much heat though.
  - kkielhofner 5y ago
    You can tweak the power limit settings for your application. In many cases you can drop the power consumption (and heat generated) while still maintaining > 90% performance but this will depend on your actual use case [0]. In my experience for many models you can reduce the power limit even further than what has been tested in this guide while barely impacting performance. [0] https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/#Power_Limiting_An_Elegant_Solution_to_Solve_the_Power_Problem https://timdettmers.com/2020/09/07/which-gpu-for-deep-learni...
  - lvl100 5y ago
    For ML? Nope. I think overheating issues are mostly for mining. I run models and 3D render quite a bit and never ran into problems.

Totally depends on your budget. The DGX A100 [1] is quite good if you have a fat wallet [1] https://www.nvidia.com/en-us/data-center/dgx-a100/ https://www.nvidia.com/en-us/data-center/dgx-a100/

atty 5y ago

Really depends on what you mean by large. If you mean truly large, you will need a cluster to train it in any reasonable amount of time. You’d probably want to look at servers built on the HGX platform (8 A100s per server). We use servers leased in bulk from traditional server providers (think Dell, HP, etc). If you mean more like “as large as personally affordable”, then you’d probably want to look at something like the RTX 3090, if you can get lucky and find it at MSRP, it has 24 gigs of memory. Nvidia also has their workstation cards with up to 48 gigs if I remember correctly, but if I were buying cards for myself, I would wait until I could get two 3090s somewhere close to MSRP, instead of paying the markup on the workstation cards (unless you want to have more than 2 in a workstation, in which case you’d need to go for those)

lvl100 5y ago

2 x 3090FE is the best bang for your buck.

cinntaile 5y ago
Do you need watercooling to keep them from running too hot?
- maxwells-daemon 5y ago
  I use 2x3090 to train large language models, and mine don't thermal-throttle with air cooling even though they're right next to each other. Eth mining does generate too much heat though.
- kkielhofner 5y ago
  You can tweak the power limit settings for your application. In many cases you can drop the power consumption (and heat generated) while still maintaining > 90% performance but this will depend on your actual use case [0]. In my experience for many models you can reduce the power limit even further than what has been tested in this guide while barely impacting performance. [0] https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/#Power_Limiting_An_Elegant_Solution_to_Solve_the_Power_Problem https://timdettmers.com/2020/09/07/which-gpu-for-deep-learni...
- lvl100 5y ago
  For ML? Nope. I think overheating issues are mostly for mining. I run models and 3D render quite a bit and never ran into problems.