18 ms·

4.5*400 is a lot lower than 700 so it’s actually a power reduction (vs using multiple A100s).

by hwers 4y ago

lunixbochs 4y ago
Sure, I think having dedicated acceleration blocks for transformers explains that pretty well, and is definitely exciting for people who can take advantage of it. I was more contextualizing this, where a potential 75% increase in power (on a smaller node) to deliver 50% more performance is less impressive: > I think it's even more impressive that the smallest increase vs. A100 is still 50% (for ResNet) If I were deploying 8-GPU nodes today, I'm not sure whether I would want to pick between 4-GPU nodes or doubling my node TDP. The DGX A100 and H100 both have 8 cards, and 640GB of VRAM, but peak power increased from 6.2kW -> 10.2kW. If you're training giant models and constrained on VRAM, you'll potentially need just as many nodes as before just to fit your models into memory. Your training will be faster but it's hard to avoid the power increase. The power increase is much more reasonable if you stick with PCIe SKUs (300W -> 350W, literally half the power draw of the SXM SKU), but you pay for that with 20% less compute and 33% less memory bandwidth.
- rajnathani 4y ago
  > I was more contextualizing this, where a potential 75% increase in power (on a smaller node) to deliver 50% more performance is less impressive: That's misleading, as the GPU power utilization during ResNet is very likely not 100% for either the A100 or H100.
  - lunixbochs 4y ago
    I'm able to trivially sustain 83% (250W / 300W) of the TDP of an Ampere A40 training a model the size of resnet50 with a basic pytorch training loop. Remember the lower-TDP PCIe H100 has 20% slower compute and 33% slower memory than the SXM model, suggesting the increased power delivery from the PCIe (350W) to SXM (700W) model is a major factor in the performance even for H100 vs H100. I don't think it's misleading to say that power is extremely likely to be a factor in the demonstrated performance increase from A100 to H100 for non-transformer workloads until proven otherwise. I don't think anyone here has a DGX H100 in hand yet to test this. Edit: wait, also I looked closer at the numbers - in MLPerf 2.1, NVIDIA only submitted results for 1x H100. There's no 1x A100 result submitted for resnet50, just an 8x A100 number, which NVIDIA seems to have divided by 8 to get a "per accelerator" number to compare to their 1x H100. That doesn't feel like a clean comparison, as you can have performance loss when scaling. I'd rather see 1v1 or 8v8.

4.5*400 is a lot lower than 700 so it’s actually a power reduction (vs using multiple A100s).

by hwers 4y ago

4.5*400 is a lot lower than 700 so it’s actually a power reduction (vs using multiple A100s).

Sure, I think having dedicated acceleration blocks for transformers explains that pretty well, and is definitely exciting for people who can take advantage of it. I was more contextualizing this, where a potential 75% increase in power (on a smaller node) to deliver 50% more performance is less impressive: > I think it's even more impressive that the smallest increase vs. A100 is still 50% (for ResNet) If I were deploying 8-GPU nodes today, I'm not sure whether I would want to pick between 4-GPU nodes or doubling my node TDP. The DGX A100 and H100 both have 8 cards, and 640GB of VRAM, but peak power increased from 6.2kW -> 10.2kW. If you're training giant models and constrained on VRAM, you'll potentially need just as many nodes as before just to fit your models into memory. Your training will be faster but it's hard to avoid the power increase. The power increase is much more reasonable if you stick with PCIe SKUs (300W -> 350W, literally half the power draw of the SXM SKU), but you pay for that with 20% less compute and 33% less memory bandwidth.

rajnathani 4y ago
> I was more contextualizing this, where a potential 75% increase in power (on a smaller node) to deliver 50% more performance is less impressive: That's misleading, as the GPU power utilization during ResNet is very likely not 100% for either the A100 or H100.
- lunixbochs 4y ago
  I'm able to trivially sustain 83% (250W / 300W) of the TDP of an Ampere A40 training a model the size of resnet50 with a basic pytorch training loop. Remember the lower-TDP PCIe H100 has 20% slower compute and 33% slower memory than the SXM model, suggesting the increased power delivery from the PCIe (350W) to SXM (700W) model is a major factor in the performance even for H100 vs H100. I don't think it's misleading to say that power is extremely likely to be a factor in the demonstrated performance increase from A100 to H100 for non-transformer workloads until proven otherwise. I don't think anyone here has a DGX H100 in hand yet to test this. Edit: wait, also I looked closer at the numbers - in MLPerf 2.1, NVIDIA only submitted results for 1x H100. There's no 1x A100 result submitted for resnet50, just an 8x A100 number, which NVIDIA seems to have divided by 8 to get a "per accelerator" number to compare to their 1x H100. That doesn't feel like a clean comparison, as you can have performance loss when scaling. I'd rather see 1v1 or 8v8.