I have heard that one of the current goals in scientific computing is to surpass the exascale level by 2018 or so... After having passed the petascale a few years ago, one would think that the exascale wouldn't be so much of a challenge... If it's theoretically possible, what challenges are there to computing at this speed? Have we already reached the limitations of moore's law?
$\begingroup$
$\endgroup$
4
-
$\begingroup$ The US Department of Energy writes entire reports and presentations on the current obstacles to reaching exascale computing. Therefore, your question is too broad. Please narrow the scope of your question. $\endgroup$– Geoff OxberryJan 15, 2012 at 4:02
-
$\begingroup$ The answer to this can still be reasonably summarized, and the majority of the community agrees on what needs to be done, I'll try to answer this fully later. $\endgroup$– Aron AhmadiaJan 15, 2012 at 9:38
-
1$\begingroup$ I quite disagree all this downvoting in the few days. It is still nice to have this question if some properly answers it. $\endgroup$– shuhaloJan 15, 2012 at 12:54
-
$\begingroup$ @Martin: I think it's a question pertinent to computational science, but like I said, quite broad. I'll try to answer it anyway, because I think it is useful to answer it. I just think it's hard to cover even all of the main points. $\endgroup$– Geoff OxberryJan 16, 2012 at 0:56
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
3
We reached the petascale through an extreme effort in concurrency. While the computational capability of modern processing cores has increased by roughly an order of magnitude in the last ten years (see Herb Sutter's excellent article "The Free Lunch is Over"), we are now capable of much more sophisticated shared-memory and distributed memory architectures, as evidenced by the success of GPUs for scientific computing and massively parallel systems such as the IBM Blue Gene.
The largest challenge to exascale systems is fundamentally financial. The K computer cost well over 1 billion dollars to develop. An exaflop machine would require similar investment.
The second largest challenge is arguably power. A system one hundred times the size of the 10 petaflop/s K machine would require 1 GW.
I think another major challenge to the utility of an exascale machine is the need for algorithms and software that are not only extremely concurrent, but largely asynchronous as well, as the costs of synchronizing millions of processing cores will eventually overwhelm any benefits of a system so powerful.
answered Jan 15, 2012 at 9:50
-
$\begingroup$ For comparison purposes on the power use, the outputs of the Aswan High Dam and Hoover Dam are about 2 GW each. So an exascale computer would require resources equivalent to a large-scale hydroelectric plant all by itself. (Kind of mind-boggling, actually.) $\endgroup$– aeismailJan 15, 2012 at 10:12
-
2$\begingroup$ And furthermore quite a waste. I would be better to invest in better algorithms rather than to invest in bigger and bigger machines which take some nuclear power plants to stay running. How much scientific staff in modelling and numerical tuning could be paid for that much money? $\endgroup$– shuhaloJan 15, 2012 at 15:18
-
$\begingroup$ @Martin: I think that what the vast majority of scientists really want is fast-turnaround (i.e., minutes instead of days) access to petascale machines, and I suspect that it won't happen until a few exascale machines are built. I am very much on the algorithmic side of things, but we need investments in both the hardware and software. $\endgroup$ Jan 15, 2012 at 21:36
$\begingroup$
$\endgroup$
I agree with Aron's answer, but I would go further.
Also, as I mentioned in my first comment, there are multiple reports and presentations out there. If you look at the US Department of Energy (DOE) Advanced Scientific Computing Research (ASCR) Programs and Reports web site, you'll see that the first 4 reports mention exascale in the title, and several reports since 2008 have mentioned computing "at the extreme scale" in the title, and include references to exascale computing. (As a side note, these reports also happen to be good for applying to government programs, because you get to see what these programs care about. Skimming some of these reports was part of the motivation for the comment.)
I don't think that the largest challenge to exascale systems is financial, given the considerable emphasis that computational scientists, and people in government, are placing on "big iron". The effort to reach petascale included a plethora of DOE reports that trumpeted petascale as the next big important thing in computational science, ostensibly for national security reasons as well as making new science possible. As noted in the Fall 2010 DOE ASCR report The Opportunities and Challenges of Exascale Computing on page 3, "the benefits outweigh the costs." There was a major funding push to reach petascale, and I don't see why the funding would dry up anytime soon.
The biggest challenge, in my opinion, is power, because that's going to be an engineering design limit that severely limits where you can place exascale computing facilities. To my knowledge, the only facilities that can supply 1 GW of power (from Aron's estimate) are large-scale hydroelectric facilities (like aeismail pointed out) or large-scale nuclear power plants (which tend to supply 1 - 1.3 GW of electricity, see this list of nuclear reactors on Wikipedia). Both sources of power have drawbacks that will make facility placement tricky.
Both power and cost are going to drive the development of entirely new technology. Estimates place exascale computing facilities constructed out of today's technology at a capital cost of \$100B just for the computer, plus operating costs of \$1B/yr to supply the power, plus its own power plant, as discussed above. If you look at the 2012 US Budget (see the graphic here for a relatively easy to read version), you'll see that the discretionary budget of the US is \$1.2T, and the DOE got around \$29.5B of that, and funding constraints are one of the reasons that we won't see an exascale system now. However, just because it's too expensive now does not mean that funding will prohibit progress, because the investment is going to focus on developing advanced computing technologies in the hopes of making them both cost-effective and energy-efficient.
The design specs that the DOE is proposing call for an exascale system to be developed by 2018 with a power requirement of 20 MW. You can find these statistics on page 48 of The Opportunities and Challenges of Exascale Computing. What's interesting about the statistics is how the factor of change the DOE would like to see in various components and metrics varies. For instance, the DOE would like a 2018 cluster to be 500 times faster than its 2012 counterpart, but require 3 times the power, and have 33 times the memory. The foreseeable increase in computing power is going to come from concurrency; the DOE plans to scale up from 12 CPU/node in a 2012 cluster to 1000+ CPU/node in a 2018 cluster. For another perspective on how to realize an exascale system using different technology, see the 2008 DARPA report Exascale Computing Study: Technology Challenges in Achieving Exascale Systems.
This scaling in the number of CPUs (and generally, in the number of components) is going to cause a number of problems. For starters, the sheer number of components is going to make it more likely that a component will fail during a supercomputing job. (For a rough metric of how many components of a particular type are expected to fail, take the time you expect your job to run, multiply it by the number of components of a given type, and divide by that component type's mean time between failures (MTBF).) Since component failure is more likely to occur due to quantity, software is going to have to be fault-tolerant. Also, as Aron noted, concurrency is going to be a big issue; if you want to take full advantage of exascale systems, you're going to have to use as many processors as you can, all the time. The increase in concurrency is going to require programmers to radically rethink how they design algorithms so that they can hide latency and avoid blocking conditions slowing down their code. For this reason, people propose designing asynchronous algorithms. Programming for concurrency is already a challenge in petascale systems, since many applications do not use more than 10-20% of the peak computing capacity of today's DOE supercomputing clusters; I know that there are some efforts focused on making it easier for developers to program more effectively in parallel (both in reducing developer time investments, and in increasing parallel performance). Memory locality is going to be an issue, since multicore systems are going to add another level in the memory hierarchy. One way to move closer to achieving the FLOPs per power benchmark is to use GPUs as part of a heterogeneous supercomputing cluster. Memory movement is going to become another potential bottleneck in parallel computing.
The last thing I can think of is data. Even on today's supercomputing systems, we already have a problem in that simulations can generate more data than we know what to do with. In combustion applications, which is where most of my experience comes from, the largest supercomputing codes just throw away large quantities of data because they are limited in how much they can store. In addition to some physical intuition, they rely on algorithms such as feature extraction to help them decide what data to keep and what data to discard. As we move into the exascale, the issue of generating more data than you can store, and how you analyze it are going to become more important.
And even after that long answer, I'm sure that I'm leaving stuff out.
answered Jan 16, 2012 at 2:21
$\begingroup$
$\endgroup$
Resilience will be the limiting factor to exascale. It is going to be relatively straightforward to build an exascale system for less than $1B that uses less than 50MW if the machine is allowed to crash every 15 minutes or compute the wrong answer most of the time.
Silent memory errors are already occurring and engineering system software and algorithms to compute accurately in spite of them is going to cost a lot of money and/or power.
The statement that money and power are the bottlenecks isn't wrong, but those are indirect problems. We need money and power to make the machine reliable; it's relatively easy to reduce the costs (power and financial) if one reduces the reliability of the system.
