6 ms·
A day in the life of the fastest supercomputer
- ungreased0675 2y agoI was hoping for a list of projects this system has queued up. It’d be interesting to see where the priorities are for something so powerful.
- dekhn 2y agoI haven't been able to find a web-accessible version of their SLURM queue, nor could I find the allocations (compute amounts given to specific groups). You can see a subset of the allocations here: https://www.ornl.gov/news/incite-program-awards-supercomputing-time-75-high-impact-projects https://www.ornl.gov/news/incite-program-awards-supercomputi...
- pelagicAustral 2y agoYou can infer a little from this [0] article: ORNL and its partners continue to execute the bring-up of Frontier on schedule. Next steps include continued testing and validation of the system, which remains on track for final acceptance and early science access later in 2022 and open for full science at the beginning of 2023. UT-Battelle manages ORNL for the Department of Energy’s Office of Science, the single largest supporter of basic research in the physical sciences in the United States. The Office of Science is working to address some of the most pressing challenges of our time. For more information, please visit energy.gov/science [0] https://www.ornl.gov/news/frontier-supercomputer-debuts-worlds-fastest-breaking-exascale-barrier https://www.ornl.gov/news/frontier-supercomputer-debuts-worl...
- deleted 2y ago[deleted]
- iJohnDoe 2y agoThe analogies used in this article were a bit weird. Two things I’ve always wondered since I’m not an expert. 1. Obviously, applications must be written to run effectively to distribute the load across the supercomputer. I wonder how often this prevents useful things from being considered to run on the supercomputer. 2. It always seems like getting access to run anything on the supercomputer is very competitive or even artificially limited? A shame this isn’t open to more people. That much processing resources seems like it should go much further to be utilized for more things.
- tryauuum 2y agoI feel like the name "supercomputer" is overhyped. It's just many normal x86 machines running Linux and connected with fast network. Here in Finland I think you can use LUMI supercomputer for free. With a condition that the results should be publically available
- bjelkeman-again 2y agoHow to get access to Lumi https://www.lumi-supercomputer.eu/get-started/ https://www.lumi-supercomputer.eu/get-started/
- NegativeK 2y agoI think you've used the "just" trap to trivialize something. I'm surprised that Frontier is free with the same conditions; I expected researchers to need grant money or whatever to fund their time. Neat.
- lokimedes 2y agoIn the beginning they were just “Beowulf clusters” compared to “real” supercomputers. Isn’t it always like this, the romantic and exceptional is absorbed by the sheer scale of the practical and common once someone discovers a way to drive the economy at scale? Cars, aircraft, long-distance communications, now perhaps AI? Yet the words may still capture the early romance.
- markstock 2y agoFYI: LUMI uses a nearly identical architecture as Frontier (AMD CPUs and GPUs), and was also made by HPE.
- msteffen 2y agoMy former employer (Pachyderm) was acquired by HPE, who built Frontier (and sells supercomputers in general), and I’ve learned a lot about that area since the acquisition. One of the main differences between supercomputers and eg a datacenter is that in the former case, application authors do not, as a rule, assume hardware or network issues and engineer around them. A typical supercomputer workload will fail overall if any one of its hundreds or thousands of workers fail. This assumption greatly simplifies the work of writing such software, as error handling is typically one of the biggest, if not the biggest, sources of complexity a distributed system. It makes engineering the hardware much harder, of course, but that’s how HPE makes money. A second difference is that RDMA (Remote Direct Memory Access—the ability for one computer to access another computer’s memory without going through its CPU. The network card can access memory directly) is standard. This removes all the complexity of an RPC framework from supercomputer workloads. Also, the L1 protocol used has orders of magnitude lower latency than Ethernet, such that it’s often faster to read memory on a remote machine than do any kind of local caching. The result is that the frameworks for writing these workloads let you more or less call an arbitrary function, run it on a neighbor, and collect the result in roughly the same amount of time it would’ve taken to run it locally.
- kaycebasques 2y agoWhat's the documentation like for supercomputers? I.e. when a researcher gets approved to use a supercomputer, do they get lots of documentation explaining how to set up and run their program? I got the sense from a physicist buddy that a lot of experimental physics stuff is shared informally and never written down. Or maybe each field has a couple popular frameworks for running simulations, and the Frontier people just make sure that Frontier runs each framework well?
- tryauuum 2y agoGoogle openmpi, mpirun, slurm. It's not complex. It's like kubernetes but invented long ago before kubernetes
- sega_sai 2y agoI know that DOE's supercomputer NERSC has a lot of documentation https://docs.nersc.gov/getting-started/ https://docs.nersc.gov/getting-started/ . Plus they also have weekly events where you can ask any questions about how the code/optimisation etc (I have never attended those, but regularly get emails about those)
- piombisallow 2y agoTake a look here if you're curious, as an example: https://docs.ncsa.illinois.edu/systems/delta/en/latest/ https://docs.ncsa.illinois.edu/systems/delta/en/latest/ 90% of my interactions are ssh'ing into a login node and running code with SLURM, then downloading the data.
- markstock 2y agohttps://docs.olcf.ornl.gov/systems/frontier_user_guide.html https://docs.olcf.ornl.gov/systems/frontier_user_guide.html This will have much of what you need.
- ok123456 2y agoYou run things more or less like you do on your Linux workstation. The only difference is you run your top-level script or program through a batch processing system on a headend node. You typically develop programs with MPI/OpenMP to exploit multiple nodes and CPUs. In Fortran, this entails a few pragmas and compiler flags.
- kkielhofner 2y agoI have a project on Frontier - happy to answer any questions! Funny story about Bronson Messer (quoted in the article): On my first trip to Oak Ridge we went on a tour of “The Machine”. Afterwards we were hanging out on the observation deck and got introduced to something like 10 people. Everyone at Oak Ridge is just Tom, Bob, etc. No titles or any of that stuff - I’m not sure I’ve ever heard anyone refer to themselves or anyone else as “Doctor”. Anyway, the guy to my right asks me a question about ML frameworks or something (don’t even remember it specifically). Then he says “Sorry, I’m sure that seems like a really basic question, I’m still learning this stuff. I’m a nuclear astrophysicist by training”. Then someone yells out “AND a three-time Jeopardy champion”! Everyone laughs. You guessed it, guy was Bronson. Place is wild.
- johnklos 2y ago> anyone refer to themselves or anyone else as "Doctor". Reminds me of the t-shirt I had that said, "Ok, Ok, so you've got a PhD. Just don't touch anything."
- dmd 2y agoI think when I walked back into my defense and they said "congratulations, Doctor Drucker" was the last time anyone ever called me Doctor except for possibly a hotel clerk when I selected 'Dr' as my honorific. It's just not in the culture, assuming you mostly work among other PhDs.
- kkielhofner 2y agoGrowing up my dad was a very well known PhD in his field (Occupational Therapy). There were quite a few people who insisted on using Doctor when referring to others, calling themselves Doctor, etc. He never did but I experienced it quite a bit.
- dmd 2y agoI'm talking about scientific/research PhDs, not medical. The title is absolutely used in medical.
- 7373737373 2y agoSo what is the actual utilization % of this machine?
- nradclif 2y agoI don’t know the exact utilization, but most large supercomputers that I’m familiar with have very high utilization, like around 90%. The Slurm/PBS queue times can sometimes be measured in days.
- wickberg 2y agoOn a node-level, usually these are aiming for around 90-95% allocated. Note that, compared to most "cloud" applications, that usually involves a number of tricks at the system scheduling level to achieve. At some point, in order to concurrently allocate a 1000-node job, all 1000 nodes will need to be briefly unoccupied ahead of that, and that can introduce some unavoidable gaps in system usage. Tuning in the "backfill" scheduling part of the workload manager can help reduce that, and a healthy mix of smaller single-node short-duration work alongside bigger multi-day multi-thousand-node jobs helps keep the machine busy.
- dauertewigkeit 2y agoDon't the industry labs have bigger machines by now? I lost track.
- Mistletoe 2y agoNot any that we know about. https://top500.org/lists/top500/list/2024/06/ https://top500.org/lists/top500/list/2024/06/
- wickberg 2y ago"Aurora" at Argonne National Labs is intended to be a bit bigger, but has suffered through a long series of delays. It's expected to surpass Frontier on the TOP500 list this fall once they some issues resolved. El Capitan at LLNL is also expected to be online soon, although I'm not sure if it'll be on the list this fall or next spring. As others note, these systems are measured by running a specific benchmark - Linpack - and require the machine to be formally submitted. There are systems in China that are on a similar scale, but, for political reasons, have not formally submitted results. There are also always rumors around the scale of classified systems owned by various countries that are also not publicized. Alongside that, the hyperscale cloud industry has added some wrinkles to how these are tracked and managed. Microsoft occupies the third position with "Eagle", which I believe is one of their newer datacenter deployments briefly repurposed to run Linpack. And they're rolling similar scale systems out on a frequent basis.
- cubefox 2y ago> With its nearly 38,000 GPUs, Frontier occupies a unique public-sector role in the field of AI research, which is otherwise dominated by industry. Is it really realistic to assume that this is the "fastest supercomputer"? What are estimated sizes for supercomputers used by OpenAI, Microsoft, Google etc? Strangely enough, the Nature piece only mentions possible secret military supercomputers, but not ones used by AI companies.
- rcxdude 2y agoThere is a difference between a supercomputer and just a large cluster of compute nodes: mainly this is in the bandwidth between the nodes. I suspect industry uses a larger number of smaller groups of highly-connected GPUs for AI work.
- p1esk 2y agoDo you mean this supercomputer has slower internode links? What are its links? For example, xAI just brought up 100k GPU cluster, most likely with 800Gbps internode links, or maybe even double that. I think the main difference is in the target numerical precision: supercomputers such as this one focus on maximizing FP64 throughput, while GPU clusters used by OpenAI or xAI want to compute in 16 or even 8 bit precision (BF16 or FP8).
- markstock 2y agoEach node has 4 GPUs, and each of those has a dedicated network interface card capable of 200 Gbps each way. Data can move right from one GPU's memory to another. But it's not just bandwidth that allows the machine to run so well, it's a very low-latency network as well. Many science codes require very frequent synchronizations, and low latency permits them to scale out to tens of thousands of endpoints.
- p1esk 2y ago200 Gbps Oh wow, that’s pretty bad.
- langcss 2y agoOr worlds smallest cloud provider?
- CaliforniaKarl 2y agoThat's a bit of an apples-and-oranges comparison. Cloud services normally have different design goals. HPC workloads are often focused on highly-parallel jobs, with high-speed and (especially) low-latency communications between nodes. Fun fact: In the NVIDIA DGX SuperPOD Reference Architecture, each DGX H100 system (which has eight H100 GPUs per system) has four Infiniband NDR OSFP ports dedicated to GPU traffic. IIRC, each OSFP port operates at 200 Gbps (two lanes of 100 Gbps), allowing each GPU to effectively have its own IB port for GPU-to-GPU traffic. (NVIDIA's not the only group doing that, BTW: Stanford's Sherlock 4.0 HPC environment[2], in their GPU-heavy servers, also uses multiple NDR ports per system.) Solutions like that are not something you'll typically find in your typical cloud provider. Early cloud-based HPC-focused solutions centered on workload locality, not just within a particular zone but with a particular part of a zone, with things like AWS Placement Groups[3]. More-modern Ethernet-based providers will give you guides like [4], telling you how to supplement placement groups with directly-accessible high-bandwidth network adapters, and in particular support for RDMA [4] or RoCE (RDMA over Converged Ethernet), which aims to provide IB-like functionality over Ethernet. IMO, the closest analog you'll find in the cloud, to environments like Frontier, is going to be IB-based cloud environments from Azure HPC ('general' cloud) [5] and specialty-cloud folks like Lambda Labs [6]. [1]: https://docs.nvidia.com/dgx-superpod/reference-architecture-scalable-infrastructure-h100/latest/abstract.html https://docs.nvidia.com/dgx-superpod/reference-architecture-... [2]: https://news.sherlock.stanford.edu/publications/sherlock-4-0-a-new-cluster-generation https://news.sherlock.stanford.edu/publications/sherlock-4-0... [3]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placemen... [4]: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html [5]: https://azure.microsoft.com/en-us/solutions/high-performance-computing https://azure.microsoft.com/en-us/solutions/high-performance... [6]: https://lambdalabs.com/nvidia/dgx-systems https://lambdalabs.com/nvidia/dgx-systems
- johnklos 2y agoThe world's smallest cloud provider could be someone running a single Raspberry Pi Zero. "Cloud" doesn't mean much more than "computer connected to the Internet".
- physicsguy 2y agoFastest publicly known supercomputer…
- nxobject 2y agoI'm curious – how much do classified projects play into the workload of Frontier?
- wickberg 2y agoFrontier runs unclassified workloads. Other Department of Energy systems, such as the upcoming "El Capitan" at LLNL (a sibling to Frontier, procured under the same contract) are used for classified work.