6 ms·

A supercomputer comparable to mid-size hyperscaler DC. (and no, it doesn't have uniquely good interconnect, it's broadly on par with the HPC GPU instances avail

A supercomputer comparable to mid-size hyperscaler DC. (and no, it doesn't have uniquely good interconnect, it's broadly on par with the HPC GPU instances available from AWS and Azure)

ClumsyPilot 4y ago
Whats is the difference between a DC and why don't DCs appear in supercomputer rankings?
- freemint 4y ago
  Optimization for different workloads, scheduling is per workload not renting per machine and that they are submitted the TOP500 list and run benchmarks. Why do not DCs appear? Because they have not submitted benchmarks and power measurements.
  - SiempreViernes 4y ago
    > scheduling is per workload This is really the key: a supercomputer has the (software) facilities that makes it possible to launch one coordinate job that runs across all nodes. A data centre is just a bunch of computers placed next to each other, with no affordances to coordinate things across them. At on point in time the hardware differences were much greater between the two, but the fundamental distinction where a supercomputer really is concerned with having the ability to be "one" computer remains.
    - zekrioca 4y ago
      Indeed, and that’s why it is called a “supercomputer”.
- wongarsu 4y ago
  I think the main difference is that on a supercomputer you generally run one task at a time, while in a DC you have computers that do different, unrelated things. The rest kind of follows from that, like how a supercomputer that consists of multiple computers needs a fast, low-latency interconnect between them to coordinate and exchange results, while computers in a DC care a lot less about each other. On the other hand the distinction is fluid. Google could call the indexers that power their search engine a supercomputer, but they prefer to talk about datacenters
  - SiempreViernes 4y ago
    Not so much "generally" as having the ability of doing it, but it is true that it a supercomputer is managed like it is one big thing that has one job queue it tries to optimise.
- why_only_15 4y ago
  Generally speaking a DC is designed for doing a bunch of different things that have less punishing interconnect needs, whereas supercomputers are designed for doing fewer things with higher interconnect needs. Datacenters often look like rows upon rows of racks with weaker interconnects between them, whereas supercomputers are much more tightly bound and built to work together.
- jeffbee 4y ago
  A cloud datacenter is about 50x larger than this, for starters.
- pbsd 4y ago
  Entries #13, #36, #37, #38, #39 on the current list are Azure clusters. #52 is an EC2 cluster.
  - zekrioca 4y ago
    Highly optimized one-time bare-metal runs, i.e., not a regular run using vCPUs and VMs like it would be for 99th of clients running HPC workloads over these platforms. Only to pop up in the Top500 charts and market about it. That’s all.
- dekhn 4y ago
  Because if you tried to run the supercomputer benchmark on a DC, you'd get a low score, and you can't easily make up for that by adding more computers to a DC. To win the supercomputer benchmarks, you need low-latency, high bandwidth networks that allow all the worker nodes in the computer to communicate calculation results. Different real jobs that run on supercomputers have different communications needs but none of them really scale well enough to be economic to run on datacenter style machines. What's interesting is that over time, the datacenter folks ended up adding supercomputers to their datacenters, with very large and fast database/blob storage/data warehousing systems connected up to "ML supercomputers" (like supercomputers, but typically only do single precision floating point). The two work well together so long as you scale the bandwidth between them. At the end of the day, any interesting data center has obscenely complex networking technology. For example, TPUs are PCI-attached devices in Google data centers; they plug into server machines just like GPUs. The TPUs themselves have networking between TPUs, that allows them to move important data, like gradients, between TPUs, as needed to do gradient descent and other operations, but the hosts that the TPUs are plugged into have their own networks. The TPUs form a mesh- the latest TPUs form a 3D mesh, but physically implemented through a complex optical switch, while the hosts they are attached to multiple switches which themselves from complex graphs of networking elements. When running ML, part of your job might be using the host CPU to read in training data and transform it, keeping the network busy, keeping some remote disk servers busy, while pushing the transformed data into the TPUs, which then communicate internal data between themselves and other TPUs, over an entirely distinct network. Crazy stuff.
hash07e 4y ago
Nope. It has Slinghshot-11[1] as interconnection having a raw power of 200GB speed, plus caching and other heavy optimizations. It is not only the gpu instances but the way it interconnects. This model has even containers available for use.[2] It is more open. [1] - https://www.nextplatform.com/2022/01/31/crays-slingshot-interconnect-is-at-the-heart-of-hpes-hpc-and-ai-ambitions/ https://www.nextplatform.com/2022/01/31/crays-slingshot-inte... [2] - https://www.lumi-supercomputer.eu/may-we-introduce-lumi/ https://www.lumi-supercomputer.eu/may-we-introduce-lumi/
slizard 4y ago
Hard no. Amazon EFA can barely come close to a dated HPC interconnect from the lower part of the top500 (when it comes to running code that does use the network, e.g. molecular dynamics or CFD), Azure does offer Cray XC or CS (https://azure.microsoft.com/en-us/solutions/high-performance-computing/#overview https://azure.microsoft.com/en-us/solutions/high-performance...) which can/will be set up as proper HPC machines with fast interconnects, but I doubt these can be readily rented in the 100s of PFlops size. Check these talks from the recent ISC EXACOMM workshop if you want to see why HPC machines and HPC computing are an entirely different league compared to traditional data center computing: https://www.youtube.com/watch?v=9PPGvqvWW8s&list=WL&index=9&t=12s https://www.youtube.com/watch?v=9PPGvqvWW8s&list=WL&index=9&... https://www.youtube.com/watch?v=q4LkF33YMJ4&list=WL&index=7 https://www.youtube.com/watch?v=q4LkF33YMJ4&list=WL&index=7

A supercomputer comparable to mid-size hyperscaler DC. (and no, it doesn't have uniquely good interconnect, it's broadly on par with the HPC GPU instances available from AWS and Azure)

ClumsyPilot 4y ago

Whats is the difference between a DC and why don't DCs appear in supercomputer rankings?

freemint 4y ago
Optimization for different workloads, scheduling is per workload not renting per machine and that they are submitted the TOP500 list and run benchmarks. Why do not DCs appear? Because they have not submitted benchmarks and power measurements.
- SiempreViernes 4y ago
  > scheduling is per workload This is really the key: a supercomputer has the (software) facilities that makes it possible to launch one coordinate job that runs across all nodes. A data centre is just a bunch of computers placed next to each other, with no affordances to coordinate things across them. At on point in time the hardware differences were much greater between the two, but the fundamental distinction where a supercomputer really is concerned with having the ability to be "one" computer remains.
  - zekrioca 4y ago
    Indeed, and that’s why it is called a “supercomputer”.
wongarsu 4y ago
I think the main difference is that on a supercomputer you generally run one task at a time, while in a DC you have computers that do different, unrelated things. The rest kind of follows from that, like how a supercomputer that consists of multiple computers needs a fast, low-latency interconnect between them to coordinate and exchange results, while computers in a DC care a lot less about each other. On the other hand the distinction is fluid. Google could call the indexers that power their search engine a supercomputer, but they prefer to talk about datacenters
- SiempreViernes 4y ago
  Not so much "generally" as having the ability of doing it, but it is true that it a supercomputer is managed like it is one big thing that has one job queue it tries to optimise.
why_only_15 4y ago
Generally speaking a DC is designed for doing a bunch of different things that have less punishing interconnect needs, whereas supercomputers are designed for doing fewer things with higher interconnect needs. Datacenters often look like rows upon rows of racks with weaker interconnects between them, whereas supercomputers are much more tightly bound and built to work together.
jeffbee 4y ago
A cloud datacenter is about 50x larger than this, for starters.
pbsd 4y ago
Entries #13, #36, #37, #38, #39 on the current list are Azure clusters. #52 is an EC2 cluster.
- zekrioca 4y ago
  Highly optimized one-time bare-metal runs, i.e., not a regular run using vCPUs and VMs like it would be for 99th of clients running HPC workloads over these platforms. Only to pop up in the Top500 charts and market about it. That’s all.
dekhn 4y ago
Because if you tried to run the supercomputer benchmark on a DC, you'd get a low score, and you can't easily make up for that by adding more computers to a DC. To win the supercomputer benchmarks, you need low-latency, high bandwidth networks that allow all the worker nodes in the computer to communicate calculation results. Different real jobs that run on supercomputers have different communications needs but none of them really scale well enough to be economic to run on datacenter style machines. What's interesting is that over time, the datacenter folks ended up adding supercomputers to their datacenters, with very large and fast database/blob storage/data warehousing systems connected up to "ML supercomputers" (like supercomputers, but typically only do single precision floating point). The two work well together so long as you scale the bandwidth between them. At the end of the day, any interesting data center has obscenely complex networking technology. For example, TPUs are PCI-attached devices in Google data centers; they plug into server machines just like GPUs. The TPUs themselves have networking between TPUs, that allows them to move important data, like gradients, between TPUs, as needed to do gradient descent and other operations, but the hosts that the TPUs are plugged into have their own networks. The TPUs form a mesh- the latest TPUs form a 3D mesh, but physically implemented through a complex optical switch, while the hosts they are attached to multiple switches which themselves from complex graphs of networking elements. When running ML, part of your job might be using the host CPU to read in training data and transform it, keeping the network busy, keeping some remote disk servers busy, while pushing the transformed data into the TPUs, which then communicate internal data between themselves and other TPUs, over an entirely distinct network. Crazy stuff.

hash07e 4y ago

Nope. It has Slinghshot-11[1] as interconnection having a raw power of 200GB speed, plus caching and other heavy optimizations. It is not only the gpu instances but the way it interconnects. This model has even containers available for use.[2] It is more open. [1] - https://www.nextplatform.com/2022/01/31/crays-slingshot-interconnect-is-at-the-heart-of-hpes-hpc-and-ai-ambitions/ https://www.nextplatform.com/2022/01/31/crays-slingshot-inte... [2] - https://www.lumi-supercomputer.eu/may-we-introduce-lumi/ https://www.lumi-supercomputer.eu/may-we-introduce-lumi/

slizard 4y ago

Hard no. Amazon EFA can barely come close to a dated HPC interconnect from the lower part of the top500 (when it comes to running code that does use the network, e.g. molecular dynamics or CFD), Azure does offer Cray XC or CS (https://azure.microsoft.com/en-us/solutions/high-performance-computing/#overview https://azure.microsoft.com/en-us/solutions/high-performance...) which can/will be set up as proper HPC machines with fast interconnects, but I doubt these can be readily rented in the 100s of PFlops size. Check these talks from the recent ISC EXACOMM workshop if you want to see why HPC machines and HPC computing are an entirely different league compared to traditional data center computing: https://www.youtube.com/watch?v=9PPGvqvWW8s&list=WL&index=9&t=12s https://www.youtube.com/watch?v=9PPGvqvWW8s&list=WL&index=9&... https://www.youtube.com/watch?v=q4LkF33YMJ4&list=WL&index=7 https://www.youtube.com/watch?v=q4LkF33YMJ4&list=WL&index=7