I am looking for credible references stating how much resources supercomputers spend on coordinating versus doing actual task-related work. Resources could be available processing power but even Watts seem like a valid unit.

I believe one of my professors or text books once said that in massively parallel systems, up to half of the available processing power is spent on coordinating the task and message passing. Unfortunately, I can't seem to find this reference or any other material about this proportion.

I realize this will differ a lot depending on the supercomputer architecture and modern implementations probably are more efficient in this regard, so an overview of this metric across multiple architectures or evolutions (before and after dedicated message passing hardware) would be even better.

• You could easily get any number you want by choosing an appropriate computer, algorithm, and implementation. – David Ketcheson Feb 23 '16 at 6:38

A long standing favorite benchmark in high performance computing has been the HPLinpack benchmark, which measures the speed of a computer system in floating point operations per second while solving a very large, dense, linear system of equations. It is assumed that the solution takes $2/3n^{3}+2n^{2}$ floating point operations and the tester is allowed to vary $n$ to achieve maximum performance.

The benchmark measures include RPEAK (the theoretical maximum number of floating point operations per second for the system) and RMAX (the maximum achieved number of operations per second in the HPLinpack benchmark.)

It's typical for RPEAK to be a substantial fraction of RMAX, indicating that on this benchmark task, current supercomputers can achieve a significant fraction of their theoretical peak performance. For example, in the November 2015 TOP500 supercomputer rankings, the fastest machine, Tianhe-2, has RPEAK=54.902 petaflops and RMAX=33.863 petaflops.

However, the HPLinpack benchmark is widely viewed as not being representative of current workloads. HPlinpack results typically overstate the performance of supercomputers in actual applications by a large factor.

A new benchmark, called HPCG, is under development. This benchmark involves operations commonly performed in iterative methods for the solution of large sparse systems of equations arising from discretized PDE's. This workload is far more challenging for high performance computers. It's also much more representative of what supercomputers are used for in practice.

Some early results from HPCG are coming in at less than 5% of RPEAK. For example, Tianhe-2 has RPEAK=54.902 petaflops and HPCG at 0.58 petaflops (see reference below to a presentation on HPCG.)

The TOP500 HPLinpack benchmarks can be found at:

http://www.top500.org/

A presentation on HPCG can be found at:

• I was curious where the $2/3n^3 + 2n^2$ flops spec came from and had to look it up. For anyone else that's curious, it's the operation count for LU decomposition with partial pivoting, i.e. a method for solving a dense system. – Aurelius Feb 22 '16 at 20:06