Exascale computer today

Question

There is a lot of talk about exascale computing these days, and whether we will be able to reach that goal by 2018, 2019 or whatever.

I have what is probably a naive question. What are the issues with doing it now?

Specifically, today we have the AMD Radeon 295x2 It has a computing power of 11.5 TFLOPS. Combining a hundred thousand of them would give us 1.15 EFLOPS.
The power consumption of each card is slightly under 500 W, so the total consumption of all of them would be 50 MW (there would probably be some more for cooling etc). I'm only guessing, but lets say all the other stuff (cooling and whatever else) took 20 MW. Lets say the price of electric power is 60$/MWh, that would amount to slightly more than 35 million \$ per year. Price of a single graphics card is 1500 \$, which means the hardware would cost 150 million \$. Lets say infrastructure costs another 50 million \$.

Compare this to current fastest supercomputer Tianhe-2. It cost 390 million \$ to make, uses 17.6 MW (24 MW with cooling) and has a processing power of 33.86 PFLOPS.

So:

Tianhe-2
390 million \$
24 MW
33.86 PFLOPS

AMD Radeon 295x2 x 100000
200 million \$
70 MW
1.15 EFLOPS

So for the cost of the Tianhe-2, you could build a computer that is more than 30 times faster and have its running costs covered for more than 5 years. I guess that after 5 years supercomputer mostly become obsolete anyway, so you would build another one :)

What am I missing here?

Is there a difference between the floating point operation that are done by today's supercomputers and these GPUs?
Is the problem with AMD not being able to produce/supply 100,000 units of 295x2?
Are there some other practical concerns, like inability to connect 100,000 units into something that would be useful, or inability to cool them properly?
Would AMD Radeons be for some reason unstable or unreliable?

Answer 1

What am I missing here?

Most of the broader issues with your proposal are covered in What are the current obstacles to reaching exascale computing?.

I think the cost and power analysis you've done is a lower bound at best: you've calculated the cost it would take to buy 100,000 GPUs, and you can't run anything on a GPU that isn't plugged into anything.

Operating systems are typically run on CPUs, not GPUs, so for every node in your system, in addition to one (or more) GPU accelerators, you'll need a mainboard with a CPU and some RAM. Furthermore, you've mentioned nothing about interconnects, on node storage, or storage for your entire cluster. All of these things cost money and power, and that's not even counting other necessary components (e.g., cases/racks, cooling fans, heat exchangers for a water cooled system).

Is there a difference between the floating point operation that are done by today's supercomputers and these GPUs?

As far as I can tell, the main difference between a CPU and a GPU is that GPUs are generally built to execute blocks of the same operation on different data across a core, and branching has poor performance. Beyond that, there's really no high-level difference. Some of today's supercomputers use GPUs (for instance, Titan), so I don't think there's a whole lot of difference into you start looking at low-level details.

Is the problem with AMD not being able to produce/supply 100,000 units of 295x2?

I doubt that.

Are there some other practical concerns, like inability to connect 100,000 units into something that would be useful, or inability to cool them properly?

Connecting the units isn't an issue. Cooling probably isn't an issue, if you can find the power, and water (if necessary), but it would be expensive. The main practical concern would be reliability (see below).

Would AMD Radeons be for some reason unstable or unreliable?

The main problem is that with so many components, all of them have to be extremely reliable to avoid nodes going down during a computation that uses the whole machine (that is, to avoid hard errors).

Soft errors (flipping a bit, for instance) also become a concern at very large scale; for instance, the lead in the solder used to attach components to the motherboards will occasionally emit a small amount of radiation that can flip a bit in memory. Sometimes, bit flips will affect an algorithm, sometimes they won't. Recovering from soft errors is an area of active research.

Answer 2

In addition to Geoff's great points:

Single versus double precision

The Radeon's quoted performance is single precision, but HPC benchmarks generally measure double precision (including the Tianhe-2 number). The Radeon has poor double precision performance, but if you buy a card focusing on double precision, expect to take at least a factor of 3 hit on performance/Watt for compute-limited operations. Many cards focusing on double precision performance will also provide ECC and bear an even higher price tag.

ECC memory

Server-grade GPUs generally cost 3-5x as much as their consumer brethren due to reliability features like ECC memory and enhanced double precision performance, along with the smaller market.

Peak versus actual performance

You quote theoretical peak for the GPU, but HPL benchmark performance for Tianhe-2, which attains only 61% of its theoretical peak. The limited graphics memory on the Radeon may further limit performance.

While it is likely that "exaflops" will eventually be declared using HPL, it is widely agreed that HPL is not representative of real applications, most of which attain less than 10% of "peak" due to a combination of memory bandwidth limitations, limited data locality, integer instructions/program logic, and irregular computation needed by the application. Efforts such as HPCG and HPGMG are attempting to create more representative benchmarks. (I am a developer of HPGMG.)