The prefix “exa” refers to the number one quintillion (1018). In the context of computing, an exascale computer can do one quintillion, or a billion billion, calculations per second.

The research group I am doing my Ph.D. in both studies and uses supercomputers. The area is sometimes also referred to as supercomputing or high-performance computing. Historically, one of the main objectives of supercomputing was to enable and accelerate computational science using really fast computers. For example, simulating galaxies, forecasting weather, or even modeling the movement of molecules requires enormous amounts of calculations, which wouldn’t be possible on personal computers (at least not if we wanted the calculations to finish within our lifetimes).

In this blog, I will recount how the abilities and use cases of supercomputers have evolved over the last few decades. I will then discuss how I approach and think about supercomputing in modern day as it relates to my dissertation research in machine learning. I will conclude by laying the groundwork for a future vision in which computational science remains open and serves the public good.

This is a card from the IBM Blue Gene line of supercomputers that I was given in the first summer of my Ph.D. I have long since lost the description of which specific supercomputer it was a part of, but after some online sleuthing1, I believe this came from the BlueGene/P machine at Argonne National Laboratory. This class of machine debuted in 2007 as the fastest supercomputer of its time and was the first to break the petaflop computing barrier2. It could do one quadrillion (that's 15 zeros!) operations per second. This machine has been decommissioned for a while, but over its lifetime it enabled computational workflows at scales never before imagined. The machines I work on today have long surpassed the petaflop scale and have recently broken the exaflop barrier: I work on systems that can do one quintillion operations per second.

So what is a (modern) supercomputer?

Seymour Cray, “the father of supercomputing,” liked to ask:

“If you were plowing a field, which would you rather use? Two strong oxen or 1024 chickens?”

Cray began his career making the case for the oxen3: he argued that by making one ultra fast computer (oxen) one could out-perform several slower computers (chickens) working together. And he did alright with this bet…for a while. But by the mid 1990s, the oxen argument was falling apart.

It turns out that if you need to do a bunch of calculations as fast as possible, multiple chickens computers working together is the most simple and efficient solution. This is because with multiple computers you can divide and conquer the computations: computations that don’t rely on each other can be run at the same time on smaller computers (called nodes) in parallel rather than waiting to be run one at a time on one big computer.

Today, the chicken argument has won. All modern supercomputers are designed to be parallel machines. During my Ph.D., I have seen large scale computers come online, be decommissioned, and proposed for future planning. One of the most notable unveilings happened in 2023 when Aurora4 came online. Aurora debuted as the world’s second exascale computer, meaning that it could do a quintillion calculations per second. This meant that the world had gone from petascale (think IBM Blue Gene) to exascale in less than two decades…in other words, we can now do 1,000 trillion more computations per second than previous-generation supercomputers.

My Adventures in Supercomputing

I was lucky enough to be one of the first researchers on Aurora. At the time, I was working on a large-scale machine learning research project in which I was studying how neural networks can work together to learn over distributed datasets. I used Aurora to train tens of thousands of neural networks to understand how information was learned and propagated. Compared to my previous projects, the size of my experiments increased by orders of magnitude overnight. And I wasn’t the only one: Aurora was being used to accelerate computational research on fusion energy and plasma physics, materials science, cosmology, and nanotechnology, to name a few.

I quickly realized that as the scale of my experiments increased, so did the problems. The speed at which parallel processes were trying to run computations and save the results raised two very specific issues:

  1. When you try to save ALL of your parallel experimental results across hundreds of thousands of experiments through a single node that is coordinating the experiments, it can cause massive slowdowns.
  2. When tons of computations are being run on a node, it generates heat. And when a node gets too hot, it gets fried.

I had been warned about the difficulties of parallel debugging in my undergraduate course work; I had even debugged parallel code on my laptop before. But never before had I done it at such a large scale on a brand new system.

Luckily, some colleagues came to my aid! With a little elbow grease, I learned how to refactor my experiments to make better use of Aurora’s parallelism: this meant that any experiments that were not dependent on each other had to be runnable (and savable!) at the same time across many nodes, and even within an experiment, any procedures that were independent were designed to be runnable at the same time on a single node across multiple GPUs. These simple tweaks solved problem #1 because now I was able to save information from many different nodes rather than being bottlenecked by one.

Problem #2 quickly resolved itself as the kinks in Aurora’s cooling system got worked out by the amazing researchers and staff who built and maintained the machine. Turns out, Aurora was one of the first large-scale public machines to use a liquid cooling system. This meant there were additional engineering challenges when designing and implementing this cooling system, since Aurora has a massive power draw, but it ended up being substantially more efficient and effective than traditional fan-based air cooling.

Today, thanks to the amazing assistance that the Aurora staff provide scientists in adapting their experimental code, and their fast response to any hardware issues, Aurora has become an amazing resource for computational science workloads.

Compute as Public Infrastructure

Aurora is a public resource: scientists from all over the country can petition to use the machine for their research. And the research these scientists do advances the public understanding of important topics like diseases, extreme weather, and (like my work) machine learning to name a few. Without supercomputers, a lot of this research would not be possible.

Much like investment in any other public infrastructure (e.g., bridges and roads), investment in a public computing infrastructure, through projects like Aurora, is a must. Private organizations are racing to build huge computational systems5, largely in response to the AI race6; these projects are often undertaken without soliciting input from the public and often harm local communities7. To counter this, it is important to continue to prioritize publicly funded supercomputers which are used for open science and designed to serve the interests of the constituents who funded them. This includes having conversations about the types of computational research that will make the world a better place and also conversations about where and how supercomputers should be built.

One of the most important topics in supercomputing is sustainability. It is no secret that supercomputers use a LOT of electricity8. And with the growing demand for (generative) AI, supercomputers are being used to both train and serve AI models to the masses. This is quickly making AI workloads one of the highest contributors to greenhouse gas emissions9.

For example, I estimate that the experiments we ran on a supercomputer for one of my research papers10, studying unwanted behavior in language models, used 16815 kWh of energy and 6437 Kg of carbon (Appendix A.6). That is about 1.6x more energy than the average American household11 uses per year! To offset the carbon footprint of my research, I have made my experimental results publicly available12 to ensure that other researchers (and even the public) can benefit from the results without having to rerun everything from scratch. Additionally, a central objective of my research agenda is to make AI models (and AI research) more sustainable; for example, in this project, I designed and used a suite of tiny language models which have a much smaller carbon footprint to test initial ideas before validating on (giant) full-sized language models. If I had used full-sized language models for all the experiments, we would have used an estimated 138,642 kWH (~13 yearly average American households of electricity)!

Today, as AI grows in popularity as a technology, it is quickly becoming one of the largest consumers of computational resources in the world. The computing infrastructure needed to satiate the projected demand for AI far supersedes current supply:

Electric power demanded by data centers is going up. Figure source: What are data centers?13

I am personally trying to make strides in ensuring that the carbon footprint of my AI research is justified and that my research itself paves the way for more efficient AI model development. However, to make a sizable impact, we need to have community-level conversations about how to design more efficient supercomputers, power them with long-term energy solutions13, and develop AI lifecycles with lower carbon footprints.

Where should the future of computing take us?

Historically, computing has led to breakthroughs in cosmology14, vaccine design, cancer research15, tsunami16 and earthquake modeling17, epidemiology18, nuclear physics19, weather forecasting20, and more21. I envision a future where this continues. To enable future scientific breakthroughs, it will be important to invest in education. The next generation of the workforce needs to be equipped to design next generation computers, know how to maintain them, and have the skills to create and critically evaluate new computational modeling technologies that run on them (e.g., AI).

Undeniably, AI is the cornerstone of almost any conversation on computing at the moment and has found itself broadcast into the public light. Importantly, as demand for AI tools grows, the researchers who build and study this technology, like myself, bear a responsibility to consider its impact on the environment, security concerns, displacement of human labor, and its role in the sciences (to name a few hot topics).

As a society, it is imperative to plan for a future in which everyone has a voice. Conversations around computing should invite public participation so that we can best prioritize computational workflows, and the underlying computational infrastructure, that advance collective societal goals.

Thanks to Francesa Vescia, Sahil Sethi, Jay Sakarvadia, and Ian Foster who provided feedback on this blog post. Generative AI was not used to draft, write, or edit this blog post. Feel free to email me if you have any thoughts about what I have written (or if u find a typo).

A note: A lot of the supercomputers and corresponding scientific breakthroughs that I reference are in the USA simply because those are the facilities and discoveries I am most familiar with. However, inspiring and amazing computational science happens all over the world. To learn more about the world’s fastest (publicly disclosed) machines see: top500.org

References

[1] Hisanobu Tomari. “IBM BlueGene/L/P/Q Compute Cards.” Okqubit.Net.

[2] “IBM Triples Performance of World’s Fastest, Most Energy-Efficient Supercomputer.” Argonne National Laboratory, 26 June 2007.

[3] “The Harness Problem.” Snormore, 21 Feb. 2026.

[4] “Frontier Remains No. 1 In the TOP500 but Aurora With Intel’s Sapphire Rapids Chips Enters With a Half-Scale System at No. 2 | TOP500.” Top500.Org.

[5] “Data centers in the United States” Cleanview, 2026.

[6] Volz, Dustin, and Cade Metz. “In A.I. Race, U.S. Models Have the Edge, but China Has Other Advantages.” The New York Times, 23 Sept. 2026.

[7] Mineo, Liz. “Why Are Communities Pushing Back Against Data Centers?” Harvard Gazette, 9 Apr. 2026.

[8] “N.J. electric bills are spiking — and AI data centers are to blame, new report says” Nj.Com, 31, March. 2026.

[9] Zewe, Adam. “Explained: Generative AI’s Environmental Impact.” MIT News, Massachusetts Institute of Technology, 17 Jan. 2025.

[10] Sakarvadia, Mansi, et al. “Mitigating memorization in language models.” International Conference on Learning Representations. Vol. 2025. 2025.

[11] U.S. Energy Information Administration. “Electricity Use in Homes.” Www.Eia.Gov, 9 May 2019.

[12] Sakarvadia, Mansi, et al. “Mitigating memorization in language models.” https://mansisak.com/memorization/. 2025.

[13] Chien. “What Are Data Centers, and Why Do They Need so Much Power?” The Institute for Climate and Sustainable Growth, 13 Oct. 2026.

[14] “Six Ways Supercomputing Advances Our Understanding of the Universe - NASA.” NASA, 15 Nov. 2024.

[15] “AI-Driven Cancer Treatment and Vaccine Development.” Argonne National Laboratory, 2026.

[16] Lawrence Livermore National Laboratory. “Big Ideas Lab Explores How Supercomputing Could Transform Tsunami Forecasting.” Lawrence Livermore National Laboratory, 27 Oct. 2026.

[17] Nemire, Brad. “Share Your Science: Predicting Earthquakes With Supercomputers.” NVIDIA Technical Blog, Dec. 2015.

[18] Zhang, Hui. “Epidemiological Modeling in the Exascale Era.” Computing Sciences, 8 Sept. 2023.

[19] “Supercomputing Illuminates Detailed Nuclear Structure.” Oak Ridge National Lab, 26 Feb. 2025.

[20] “About Supercomputers.” Weather.Gov, 2016.

[21] “Supercomputing.” Energy.Gov, 2022, https://www.energy.gov/topics/supercomputing.