6 ms·

Nvidia DGX GH200 Whitepaper

95 points by volta87 3y ago

tuetuopay 3y ago
Why is this called a whitepaper, as this is more of a documentation and architecture overview of the cluster? Wow a CLOS topology for networking, very innovative. Details on NVLink would be great. For example, the needs and problems solved by their custom cables seemingly required by NVLink would be worth a whitepaper. Don't get me wrong, this is still great the general public can get a glimpse into Grace Hopper. And they do a good job of simplifying while throwing around mind-boggling numbers (the NVLink bandwidth is insane, though no words on latency, crucial for remote memory access).
- syntaxing 3y ago
  Agreed, seems like an application note more than a white paper.
- mmaunder 3y ago
  > Why is this called a whitepaper, as this is more of a documentation and architecture overview of the cluster? That’s what a marketing white paper is and does. It’s not an academic paper.
  - flakiness 3y ago
    To be fair NVIDIA used to publish more detailed "white paper" for their GPUs ex. [1] and CPU textbooks like H&P [2] draws a lot of details from these. This less detailed "whitepaper" still has a scent of these old tradition. [1] https://www.nvidia.com/content/PDF/nvidia-ampere-ga-102-gpu-architecture-whitepaper-v2.pdf https://www.nvidia.com/content/PDF/nvidia-ampere-ga-102-gpu-... [2] https://www.amazon.com/Computer-Architecture-Quantitative-John-Hennessy/dp/012383872X https://www.amazon.com/Computer-Architecture-Quantitative-Jo...
  - danpalmer 3y ago
    I was always taught that “whitepapers” were this sort of thing and were distinct from academic papers. However this seems to be industry or ecosystem specific because the cryptocurrency ecosystem uses “whitepaper” to mean their academic papers, or at least their approximation of them.
- callalex 3y ago
  NVDA has spent too much time surrounded by cryptocurrency hacks that published “whitepapers” left and right with zero technical information or innovation. As they say, never get high on your own supply.
mmaunder 3y ago
The memory and bandwidth numbers are mind blowing. Going to be very hard to catch Nvidia. It’s as if competitors are going through the motions for participation prizes.
- jauntywundrkind 3y ago
  AMD has been shipping 128x lanes of PCIe 5.0 on chip. That's 0.5TBps. Getting up to 0.9TBps isn't that crazy, but having big enough fabric & switches to attach to is a huge feat. I have hope though. CXL switching is going to give the whole industry a very fresh look at interconnect fabrics, as a simpler to manage faster more direct alternative to PCIe. Should be good. Personally I worry it's flogging a dead horse, has too many constraints, but Ethernet could be rumbling into action again too maybe. The hyperscalers & others created a new LinuxFoundation group "Ultra Ethernet Scaling" to scale up much faster. Still, even at 1Tbps, that's a bunch of lanes (7x) of that ultra Ethernet you'd need to get to NVlink's 0.9TBps GPU interconnect. More radical breaks with Ethernet are needed than line speed bumps, things that can make switches easier to scale out big, if this realm of tech is to be good systems fabric. https://www.linuxfoundation.org/press/announcing-ultra-ethernet-consortium-uec https://www.linuxfoundation.org/press/announcing-ultra-ether... One interesting note on the DGX GH200 architecture that is super interesting to me is that it's inverted the connectivity relationship. Typically a system would have NIC & GPU hanging off the processor bus, and interconnect would go over that bus (maybe optimizing with p2p-dma to skip going through main memory, if it's fancy). But here? GPUs have a 0.9TBps connection to the NVswitch. If the CPU wants to talk to the cluster, it uses nvlink c2c to send the data to the gpu that then used it's nvlink connection to the NVswitch to send it out. Interesting reversal, interesting flourish, and gee it sure makes sense to me; the GPU is the thing! Also, past 256 GPUs, there are BlueField 3 devices for Ethernet or infiniband connectivity on DGX nodes. Which is a good but also pretty boring/standard smartnic based scale out strategy.
  - mmaunder 3y ago
    Thanks for the data rich comment. Very interesting re the inversion you mentioned. Much appreciated.
- 3y ago
m3kw9 3y ago
So basically 2x faster than H100
- deleted 3y ago
  [deleted]
- luc4sdreyer 3y ago
  They claim 1.1x to 7x, depending on what you're doing. The 10% to 50% is for the ~10k GPU LLM training, where the main bottleneck tends to be networking: > DGX GH200 enables more efficient parallel mapping and alleviates the networking communication bottleneck. As a result, up to 1.5x faster training time can be achieved over a DGX H100-based solution for LLM training at scale.
- kvetching 3y ago
  Was this upgrade known or is this out of left field and people that stocked up on H100s going to feel a little regret
  - wmf 3y ago
    It's been on the roadmap for a few years although there were no performance numbers. I assume GH200 is more expensive so the price/performance advantage may not be overwhelming. Worst case you order GH200s and then scalp your H100s on the used market.
tikkun 3y ago
As context: 1x dgx gh200 has 256x gh200s which each have 1x h100 and 1x grace cpu
- luc4sdreyer 3y ago
  Adding up to "1 exaFLOPS" (sparse FP8). For reference, the fastest FP64 supercomputer is the AMD-based Frontier supercomputer, at 1.1 exaFLOPS.
  - danbruc 3y ago
    Does sparse mean anything other than we can not actually do as many FP8 operations per second as we just claimed? To me it sounds like they can do X matrix operations per second on sparse matrices using Y FP8 operations per second, but instead of just saying what Y is they tell us how many FP8 operations would be required if the matrices were not sparse. Is this pure marketing bullshit or is there some logic to this? How sparse do those matrices have to be? Or am I misunderstanding this claim?
    - buildbot 3y ago
      It means a very specific sparsity pattern - 2:4, so 2 out of 4 values are not 0. It's not pure bullshit, because a matrix with 2:4 sparsity may represent more "information" than a matrix that is 50% smaller.
      - danbruc 3y ago
        Okay, yes, there is a bit more information than in a matrix with half the number of entries, namely the position of the zeros. But when it comes to the number of floating point operations, doubling that number seems at least somewhat questionable to me, they are not performing that many multiplications. On the other hand it would probably be hard if not impossible to achieve the same performance if one tried to manually exploit this sparsity and avoid the multiplications, so maybe under that angle it is not too unreasonable. But this also made me wonder, how does one use this in practice? If the matrices are not tiny, then they will probably have to be incredible sparse in order to always have at least two out of four entries zero. So does this just set some entries to zero if there are not enough of them in each group of four? Does one have to ensure this on its own, reorder rows and columns and introduce zeros where required and acceptable?
jacquesm 3y ago
I wonder how much this thing will cost, best I've been able to find so far is a 'low 8 digits' estimate in Anandtech article but nothing more specific than that. https://www.anandtech.com/show/18877/nvidia-grace-hopper-has-entered-full-production-announcing-dgx-gh200-ai-supercomputer https://www.anandtech.com/show/18877/nvidia-grace-hopper-has...
- tikkun 3y ago
  Some private cloud execs I talked with ballparked it at $15-25mm [1]. [1]: (I wrote this) https://gpus.llm-utils.org/nvidia-h100-gpus-supply-and-demand/#how-much-do-these-gpus-cost https://gpus.llm-utils.org/nvidia-h100-gpus-supply-and-deman...
LASR 3y ago
I would be interesting to know what kind of next-gen models this can train. On the LLM frontier, we’re starting to hit the limits of reasoning abilities in the current gen.
- paskjdfparwerwe 3y ago
  ... and the current generation is just an ensemble of the prev. generation.
smodad 3y ago
What's funny is that even though the DGX GH200 is some of the most powerful hardware available, there's such a voracious demand that it's not gonna be enough to quench it. In fact, this is one of those cases where I think the demand will always outpace supply. Exciting stuff ahead. I heard Elon say something interesting during the discussion/launch of xAI: "My prediction is that we will go from an extreme silicon shortage today, to probably a voltage-transformer shortage in about year, and then an electricity shortage in about a year, two years." I'm not sure about the timeline, but it's an intriguing idea that soon the rate limiting resource will be electricity. I wonder how true that is and if we're prepared for that.
- jiggawatts 3y ago
  He’s just plain wrong about the electricity usage going up because of AI compute. To a first approximation, the amount of silicon wafers going through fabs globally is constant. We won’t suddenly increase chip manufacturing a hundredfold! There aren’t enough fabs or “tools” like the ASML EUV machines for that. Electricity is used for lots of things, not just compute, and within compute the AI fraction is tiny. We’re ramping up a rounding error to a slightly larger rounding error. What will increase is global energy demand for overall economic activity as manufacturing and industry is accelerated by AIs. Anyone who’s played games like Factorio would know intuitively that the only two real inputs to the economy are raw materials and energy. Increases to manufacturing speed need matching increases to energy supply!
  - wmf 3y ago
    A wafer of H100s uses far more electricity than a wafer of [Apple] A16s though.
    - lwneal 3y ago
      An H100 uses up to 350 Watts, while an A16 has a TDP of only 8 W. But, the A16 is a smaller chip (about 108mm vs. the H100's 814mm) so you can fit more of them on a wafer. Since a wafer is 300mm in diameter, its area is 70685 mm^2, which would yield 86 H100's or 654 A16's. [1][2] However, that discounts the waste on the edges of the circular wafer, as well as the chip yield, which will both likely be worse for the larger chip [3]. But, assuming a generous 70% yield by area [4], one wafer's worth of H100s all packaged into GPUs and running full blast will use maybe 20 kilowatts, while the same wafer of A16s might use 3.6 kilowatts. Although in practice, the A16s will spend most of their time conserving battery power in your pocket, and even the H100s will spend some of their time idle. TSMC is now producing over 14 million wafers per year. At most 1.2 million of those are on the 3nm node, and not all of that production goes to GPUs. But as an upper bound, if we imagine that all of TSMC's wafers could be filled up with nothing but H100 chips, and if all of those H100 chips were immediately put to use running AI 24/7, how much additional load could it put on the power grid every year? The answer is, around 280 gigawatts, or if they were running 24/7 for a year, about 2500 terawatt-hours. That's about 10% of current world electricity consumption! So it's not completely implausible to imagine that a huge ramp-up in AI usage might have an effect on the electric grid. *edit: This assumes we're talking about the Apple A16 (ie. the difference between phone chips and GPU chips). If we're talking about the Nvidia A16 (ie. the difference between current GPU chips and last node's GPU chips) see pclmulqdq's comment. ⠀ [1] https://nanoreview.net/en/soc/apple-a16-bionic https://nanoreview.net/en/soc/apple-a16-bionic [2] https://www.techpowerup.com/gpu-specs/h100-pcie-80-gb.c3899 https://www.techpowerup.com/gpu-specs/h100-pcie-80-gb.c3899 [3] https://news.ycombinator.com/item?id=24185108 https://news.ycombinator.com/item?id=24185108 [4] https://www.extremetech.com/computing/analyst-tsmc-hitting-55-yields-on-3nm-node-for-apples-a17-bionic-m3-socs https://www.extremetech.com/computing/analyst-tsmc-hitting-5... [5] https://www.tsmc.com/english/dedicatedFoundry/manufacturing/fab_capacity https://www.tsmc.com/english/dedicatedFoundry/manufacturing/... [6] https://www.wolframalpha.com/input?i=%2814+million%29+*+%2820+kilowatts%29\\ https://www.wolframalpha.com/input?i=%2814+million%29+*+%282...*
moab 3y ago
Unfortunate that they don't mention the running times for any of the applications they benchmark (e.g., PageRank). Does anyone in the know have some idea how long this takes?

95 points by volta87 3y ago

Why is this called a whitepaper, as this is more of a documentation and architecture overview of the cluster? Wow a CLOS topology for networking, very innovative. Details on NVLink would be great. For example, the needs and problems solved by their custom cables seemingly required by NVLink would be worth a whitepaper. Don't get me wrong, this is still great the general public can get a glimpse into Grace Hopper. And they do a good job of simplifying while throwing around mind-boggling numbers (the NVLink bandwidth is insane, though no words on latency, crucial for remote memory access).

syntaxing 3y ago
Agreed, seems like an application note more than a white paper.
mmaunder 3y ago
> Why is this called a whitepaper, as this is more of a documentation and architecture overview of the cluster? That’s what a marketing white paper is and does. It’s not an academic paper.
- flakiness 3y ago
  To be fair NVIDIA used to publish more detailed "white paper" for their GPUs ex. [1] and CPU textbooks like H&P [2] draws a lot of details from these. This less detailed "whitepaper" still has a scent of these old tradition. [1] https://www.nvidia.com/content/PDF/nvidia-ampere-ga-102-gpu-architecture-whitepaper-v2.pdf https://www.nvidia.com/content/PDF/nvidia-ampere-ga-102-gpu-... [2] https://www.amazon.com/Computer-Architecture-Quantitative-John-Hennessy/dp/012383872X https://www.amazon.com/Computer-Architecture-Quantitative-Jo...
- danpalmer 3y ago
  I was always taught that “whitepapers” were this sort of thing and were distinct from academic papers. However this seems to be industry or ecosystem specific because the cryptocurrency ecosystem uses “whitepaper” to mean their academic papers, or at least their approximation of them.
callalex 3y ago
NVDA has spent too much time surrounded by cryptocurrency hacks that published “whitepapers” left and right with zero technical information or innovation. As they say, never get high on your own supply.

mmaunder 3y ago

The memory and bandwidth numbers are mind blowing. Going to be very hard to catch Nvidia. It’s as if competitors are going through the motions for participation prizes.

jauntywundrkind 3y ago
AMD has been shipping 128x lanes of PCIe 5.0 on chip. That's 0.5TBps. Getting up to 0.9TBps isn't that crazy, but having big enough fabric & switches to attach to is a huge feat. I have hope though. CXL switching is going to give the whole industry a very fresh look at interconnect fabrics, as a simpler to manage faster more direct alternative to PCIe. Should be good. Personally I worry it's flogging a dead horse, has too many constraints, but Ethernet could be rumbling into action again too maybe. The hyperscalers & others created a new LinuxFoundation group "Ultra Ethernet Scaling" to scale up much faster. Still, even at 1Tbps, that's a bunch of lanes (7x) of that ultra Ethernet you'd need to get to NVlink's 0.9TBps GPU interconnect. More radical breaks with Ethernet are needed than line speed bumps, things that can make switches easier to scale out big, if this realm of tech is to be good systems fabric. https://www.linuxfoundation.org/press/announcing-ultra-ethernet-consortium-uec https://www.linuxfoundation.org/press/announcing-ultra-ether... One interesting note on the DGX GH200 architecture that is super interesting to me is that it's inverted the connectivity relationship. Typically a system would have NIC & GPU hanging off the processor bus, and interconnect would go over that bus (maybe optimizing with p2p-dma to skip going through main memory, if it's fancy). But here? GPUs have a 0.9TBps connection to the NVswitch. If the CPU wants to talk to the cluster, it uses nvlink c2c to send the data to the gpu that then used it's nvlink connection to the NVswitch to send it out. Interesting reversal, interesting flourish, and gee it sure makes sense to me; the GPU is the thing! Also, past 256 GPUs, there are BlueField 3 devices for Ethernet or infiniband connectivity on DGX nodes. Which is a good but also pretty boring/standard smartnic based scale out strategy.
- mmaunder 3y ago
  Thanks for the data rich comment. Very interesting re the inversion you mentioned. Much appreciated.
3y ago

m3kw9 3y ago

So basically 2x faster than H100

deleted 3y ago
[deleted]
luc4sdreyer 3y ago
They claim 1.1x to 7x, depending on what you're doing. The 10% to 50% is for the ~10k GPU LLM training, where the main bottleneck tends to be networking: > DGX GH200 enables more efficient parallel mapping and alleviates the networking communication bottleneck. As a result, up to 1.5x faster training time can be achieved over a DGX H100-based solution for LLM training at scale.
kvetching 3y ago
Was this upgrade known or is this out of left field and people that stocked up on H100s going to feel a little regret
- wmf 3y ago
  It's been on the roadmap for a few years although there were no performance numbers. I assume GH200 is more expensive so the price/performance advantage may not be overwhelming. Worst case you order GH200s and then scalp your H100s on the used market.

tikkun 3y ago

As context: 1x dgx gh200 has 256x gh200s which each have 1x h100 and 1x grace cpu

luc4sdreyer 3y ago
Adding up to "1 exaFLOPS" (sparse FP8). For reference, the fastest FP64 supercomputer is the AMD-based Frontier supercomputer, at 1.1 exaFLOPS.
- danbruc 3y ago
  Does sparse mean anything other than we can not actually do as many FP8 operations per second as we just claimed? To me it sounds like they can do X matrix operations per second on sparse matrices using Y FP8 operations per second, but instead of just saying what Y is they tell us how many FP8 operations would be required if the matrices were not sparse. Is this pure marketing bullshit or is there some logic to this? How sparse do those matrices have to be? Or am I misunderstanding this claim?
  - buildbot 3y ago
    It means a very specific sparsity pattern - 2:4, so 2 out of 4 values are not 0. It's not pure bullshit, because a matrix with 2:4 sparsity may represent more "information" than a matrix that is 50% smaller.
    - danbruc 3y ago
      Okay, yes, there is a bit more information than in a matrix with half the number of entries, namely the position of the zeros. But when it comes to the number of floating point operations, doubling that number seems at least somewhat questionable to me, they are not performing that many multiplications. On the other hand it would probably be hard if not impossible to achieve the same performance if one tried to manually exploit this sparsity and avoid the multiplications, so maybe under that angle it is not too unreasonable. But this also made me wonder, how does one use this in practice? If the matrices are not tiny, then they will probably have to be incredible sparse in order to always have at least two out of four entries zero. So does this just set some entries to zero if there are not enough of them in each group of four? Does one have to ensure this on its own, reorder rows and columns and introduce zeros where required and acceptable?

jacquesm 3y ago

I wonder how much this thing will cost, best I've been able to find so far is a 'low 8 digits' estimate in Anandtech article but nothing more specific than that. https://www.anandtech.com/show/18877/nvidia-grace-hopper-has-entered-full-production-announcing-dgx-gh200-ai-supercomputer https://www.anandtech.com/show/18877/nvidia-grace-hopper-has...

tikkun 3y ago
Some private cloud execs I talked with ballparked it at $15-25mm [1]. [1]: (I wrote this) https://gpus.llm-utils.org/nvidia-h100-gpus-supply-and-demand/#how-much-do-these-gpus-cost https://gpus.llm-utils.org/nvidia-h100-gpus-supply-and-deman...

LASR 3y ago

I would be interesting to know what kind of next-gen models this can train. On the LLM frontier, we’re starting to hit the limits of reasoning abilities in the current gen.

paskjdfparwerwe 3y ago
... and the current generation is just an ensemble of the prev. generation.

smodad 3y ago

What's funny is that even though the DGX GH200 is some of the most powerful hardware available, there's such a voracious demand that it's not gonna be enough to quench it. In fact, this is one of those cases where I think the demand will always outpace supply. Exciting stuff ahead. I heard Elon say something interesting during the discussion/launch of xAI: "My prediction is that we will go from an extreme silicon shortage today, to probably a voltage-transformer shortage in about year, and then an electricity shortage in about a year, two years." I'm not sure about the timeline, but it's an intriguing idea that soon the rate limiting resource will be electricity. I wonder how true that is and if we're prepared for that.

jiggawatts 3y ago
He’s just plain wrong about the electricity usage going up because of AI compute. To a first approximation, the amount of silicon wafers going through fabs globally is constant. We won’t suddenly increase chip manufacturing a hundredfold! There aren’t enough fabs or “tools” like the ASML EUV machines for that. Electricity is used for lots of things, not just compute, and within compute the AI fraction is tiny. We’re ramping up a rounding error to a slightly larger rounding error. What will increase is global energy demand for overall economic activity as manufacturing and industry is accelerated by AIs. Anyone who’s played games like Factorio would know intuitively that the only two real inputs to the economy are raw materials and energy. Increases to manufacturing speed need matching increases to energy supply!
- wmf 3y ago
  A wafer of H100s uses far more electricity than a wafer of [Apple] A16s though.
  - lwneal 3y ago
    An H100 uses up to 350 Watts, while an A16 has a TDP of only 8 W. But, the A16 is a smaller chip (about 108mm vs. the H100's 814mm) so you can fit more of them on a wafer. Since a wafer is 300mm in diameter, its area is 70685 mm^2, which would yield 86 H100's or 654 A16's. [1][2] However, that discounts the waste on the edges of the circular wafer, as well as the chip yield, which will both likely be worse for the larger chip [3]. But, assuming a generous 70% yield by area [4], one wafer's worth of H100s all packaged into GPUs and running full blast will use maybe 20 kilowatts, while the same wafer of A16s might use 3.6 kilowatts. Although in practice, the A16s will spend most of their time conserving battery power in your pocket, and even the H100s will spend some of their time idle. TSMC is now producing over 14 million wafers per year. At most 1.2 million of those are on the 3nm node, and not all of that production goes to GPUs. But as an upper bound, if we imagine that all of TSMC's wafers could be filled up with nothing but H100 chips, and if all of those H100 chips were immediately put to use running AI 24/7, how much additional load could it put on the power grid every year? The answer is, around 280 gigawatts, or if they were running 24/7 for a year, about 2500 terawatt-hours. That's about 10% of current world electricity consumption! So it's not completely implausible to imagine that a huge ramp-up in AI usage might have an effect on the electric grid. *edit: This assumes we're talking about the Apple A16 (ie. the difference between phone chips and GPU chips). If we're talking about the Nvidia A16 (ie. the difference between current GPU chips and last node's GPU chips) see pclmulqdq's comment. ⠀ [1] https://nanoreview.net/en/soc/apple-a16-bionic https://nanoreview.net/en/soc/apple-a16-bionic [2] https://www.techpowerup.com/gpu-specs/h100-pcie-80-gb.c3899 https://www.techpowerup.com/gpu-specs/h100-pcie-80-gb.c3899 [3] https://news.ycombinator.com/item?id=24185108 https://news.ycombinator.com/item?id=24185108 [4] https://www.extremetech.com/computing/analyst-tsmc-hitting-55-yields-on-3nm-node-for-apples-a17-bionic-m3-socs https://www.extremetech.com/computing/analyst-tsmc-hitting-5... [5] https://www.tsmc.com/english/dedicatedFoundry/manufacturing/fab_capacity https://www.tsmc.com/english/dedicatedFoundry/manufacturing/... [6] https://www.wolframalpha.com/input?i=%2814+million%29+*+%2820+kilowatts%29\\ https://www.wolframalpha.com/input?i=%2814+million%29+*+%282...*

moab 3y ago

Unfortunate that they don't mention the running times for any of the applications they benchmark (e.g., PageRank). Does anyone in the know have some idea how long this takes?