Informally in our lab, we have developed 2 metrics to compare CFD solvers over the range of machines we have access to. One is called COMP, which stands for COde Machine Performance. This single number is supposed to represent the absolute performance of a given code on a given machine. It is computed, for a given run, by multiplying the number of cells per computing/processing core by the number of iterations performed and then dividing by the runtime. In an ideal situation, this number should be constant no matter the number of cores being used, the size of the grid or the duration of the run. It directly indicates on how many cells one iteration can be performed by one core in one second. By extension, the acronym COMP will be used as unit for measuring the performance of the codes. For examples, if a given run yields "3.2 k COMPs", it means the code is able to process one iteration on 3200 cells per core per second, or 3200 iterations on one cell per core per second, or any similar combination. Derived from COMP, we have obvious metrics like speed, speed-up and efficiency, which are just expressing in different ways raw performance and scaling.
The other metric, which is designed to compare the efficiency of different schemes/codes on the same machines, simply looks at the amount of CPU time required per unit of physical time simulated. Of course, this leaves many parameters out of the analysis, like the grid or the accuracy of the obtained solution. But we strive to compare runs on equivalent grid/accuracy (for example, if you compare a 4th order scheme with a 2nd order scheme, you should probably use half as many points for a similar accuracy).
What do you think of these metrics? Do they appear valid to you? Do you know or use other similar metrics to benchmark your CFD solver?
I should also add that we usually deal with explicit schemes on structured grids, although we are now starting to do some comparisons with a DG code. The reasoning might be different for unstructured grids and/or implicit schemes.