10
$\begingroup$

I am looking for credible references stating how much resources supercomputers spend on coordinating versus doing actual task-related work. Resources could be available processing power but even Watts seem like a valid unit.

I believe one of my professors or text books once said that in massively parallel systems, up to half of the available processing power is spent on coordinating the task and message passing. Unfortunately, I can't seem to find this reference or any other material about this proportion.

I realize this will differ a lot depending on the supercomputer architecture and modern implementations probably are more efficient in this regard, so an overview of this metric across multiple architectures or evolutions (before and after dedicated message passing hardware) would be even better.

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ You could easily get any number you want by choosing an appropriate computer, algorithm, and implementation. $\endgroup$
    – David Ketcheson
    Commented Feb 23, 2016 at 6:38

2 Answers 2

Reset to default
10
$\begingroup$

A long standing favorite benchmark in high performance computing has been the HPLinpack benchmark, which measures the speed of a computer system in floating point operations per second while solving a very large, dense, linear system of equations. It is assumed that the solution takes $2/3n^{3}+2n^{2}$ floating point operations and the tester is allowed to vary $n$ to achieve maximum performance.

The benchmark measures include RPEAK (the theoretical maximum number of floating point operations per second for the system) and RMAX (the maximum achieved number of operations per second in the HPLinpack benchmark.)

It's typical for RPEAK to be a substantial fraction of RMAX, indicating that on this benchmark task, current supercomputers can achieve a significant fraction of their theoretical peak performance. For example, in the November 2015 TOP500 supercomputer rankings, the fastest machine, Tianhe-2, has RPEAK=54.902 petaflops and RMAX=33.863 petaflops.

However, the HPLinpack benchmark is widely viewed as not being representative of current workloads. HPlinpack results typically overstate the performance of supercomputers in actual applications by a large factor.

A new benchmark, called HPCG, is under development. This benchmark involves operations commonly performed in iterative methods for the solution of large sparse systems of equations arising from discretized PDE's. This workload is far more challenging for high performance computers. It's also much more representative of what supercomputers are used for in practice.

Some early results from HPCG are coming in at less than 5% of RPEAK. For example, Tianhe-2 has RPEAK=54.902 petaflops and HPCG at 0.58 petaflops (see reference below to a presentation on HPCG.)

The TOP500 HPLinpack benchmarks can be found at:

http://www.top500.org/

A presentation on HPCG can be found at:

http://www.hpcg-benchmark.org/downloads/isc15/HPCG-ISC15-FINAL-SLIDES_update1.pdf

HPCG's website is at

http://www.hpcg-benchmark.org/

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ I was curious where the $2/3n^3 + 2n^2$ flops spec came from and had to look it up. For anyone else that's curious, it's the operation count for LU decomposition with partial pivoting, i.e. a method for solving a dense system. $\endgroup$
    – Aurelius
    Commented Feb 22, 2016 at 20:06
  • 3
    $\begingroup$ This doesn't seem to answer the question, as it says nothing about message passing. $\endgroup$
    – David Ketcheson
    Commented Feb 23, 2016 at 6:40
  • $\begingroup$ It partially answers the question in the sense that these benchmarks tell you how efficiently the floating point units are being used- you can subtract from one to find out how much time is being spent on everything else, which includes message passing among other things. $\endgroup$
    – Brian Borchers
    Commented Feb 23, 2016 at 14:25
6
$\begingroup$

The honest answer is that we don't know. The answer depends heavily on what is actually being run and what code the user has written. As Brian Borchers points out, there's a big difference between two benchmarks where we have all the code and supposedly know what that code is doing, but there's much disagreement about how representative this code is of what users of supercomputers are actually doing. Without detailed source code analysis and some heavy instrumentation of real codes on real machines, finding this ratio is next to impossible. There are some projects that are starting to collect data that might be able to get the community close to answering this question, but it is not at all settled.

In fact, the question isn't really even clear. If a cluster node's communication card has a processor on it which can only be used for communication, how do you count the time that this card spends idle not handling communication (nor anything else)? I.e., what counts as "available processing power"? Do we count badly written programs which have unoptimized compute and communications routines the same as optimized? What if someone uses a known anti-pattern in their code which deliberately under-utilizes the CPU? What about embarrassingly parallel programs that do not communicate at all (these do get run on supercomputers, I promise you)?

I wouldn't waste your time trying to quantify an off-the-cuff remark in a book or from your professor. These kinds of statements are there to remind us that parallel programming is hard and generally done poorly. Supercomputers also aren't perfectly designed to eliminate or optimize away all the waste.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.