Benchmarking performance in CFD: how to compare machines and codes?

Question

Informally in our lab, we have developed 2 metrics to compare CFD solvers over the range of machines we have access to. One is called COMP, which stands for COde Machine Performance. This single number is supposed to represent the absolute performance of a given code on a given machine. It is computed, for a given run, by multiplying the number of cells per computing/processing core by the number of iterations performed and then dividing by the runtime. In an ideal situation, this number should be constant no matter the number of cores being used, the size of the grid or the duration of the run. It directly indicates on how many cells one iteration can be performed by one core in one second. By extension, the acronym COMP will be used as unit for measuring the performance of the codes. For examples, if a given run yields "3.2 k COMPs", it means the code is able to process one iteration on 3200 cells per core per second, or 3200 iterations on one cell per core per second, or any similar combination. Derived from COMP, we have obvious metrics like speed, speed-up and efficiency, which are just expressing in different ways raw performance and scaling.

The other metric, which is designed to compare the efficiency of different schemes/codes on the same machines, simply looks at the amount of CPU time required per unit of physical time simulated. Of course, this leaves many parameters out of the analysis, like the grid or the accuracy of the obtained solution. But we strive to compare runs on equivalent grid/accuracy (for example, if you compare a 4th order scheme with a 2nd order scheme, you should probably use half as many points for a similar accuracy).

What do you think of these metrics? Do they appear valid to you? Do you know or use other similar metrics to benchmark your CFD solver?

I should also add that we usually deal with explicit schemes on structured grids, although we are now starting to do some comparisons with a DG code. The reasoning might be different for unstructured grids and/or implicit schemes.

Answer 1

At the end of the day, the only things that matter are wall-clock time to a chosen accuracy and Watts (or dollars) to a chosen accuracy.

For implementation and hardware performance, I like to measure in terms of memory bandwidth and flop/s per Watt or per dollar. If the performance of the code is very far below the hardware peak according to these metrics, then there is likely some implementation inefficiency. Alternatively, if even very simple benchmarks like STREAM perform well below machine peak, then the machine may have bottlenecks that reduce its realizable performance.

The efficiency of an algorithm really can't be measured by "cells per second" except within a restricted class. If you measure by that metric, I argue that you have missed the point.