Log-log parallel scaling/efficiency plots

Question

A lot of my own work revolves around making algorithms scale better, and one of the preferred ways of showing parallel scaling and/or parallel efficiency is to plot the performance of an algorithm/code over the number of cores, e.g.

where the $x$-axis represents the number of cores and the $y$-axis some metric, e.g. work done per unit of time. The different curves show parallel efficiencies of 20%, 40%, 60%, 80%, and 100% at 64 cores respectively.

Unfortunately though, in many publications, these results are plotted with a log-log scaling, e.g. the results in this or this paper. The problem with these log-log plots is that it's incredibly difficult to assess the actual parallel scaling/efficiency, e.g.

Which is the same plot as above, yet with log-log scaling. Note that now there is no big difference between the results for 60%, 80%, or 100% parallel efficiency. I've written a bit more extensively about this here.

So here's my question: What rationale is there for showing results in log-log scaling? I regularly use linear scaling to show my own results, and regularly get hammered by referees saying that my own parallel scaling/efficiency results don't look as good as the (log-log) results of others, but for the life of me I can't see why I should switch plot styles.

Answer 1

We are currently writing a paper that contains a number of comparable plots, and we more or less had the same problem. The paper is about comparing the scaling of different algorithms over the number of cores, which ranges between 1 and up to 100k on a BlueGene. The reason for using loglog-plots in this situation is the number of orders of magnitude involved. There is no way one can plot 6 orders of magnitude on a linear scale.

And indeed, when plotting the time over the number of cores in loglog, the algorithms are not very distinguishable, as you can see in the following plot.

Therefore, we decided to plot the parallel efficiency over the number of cores instead of the timing on a semilog-scale. The parallel efficiency is defined as $E_p=T_1/(p T_p)$, where $T_1$ is the time when using a single core, $T_p$ is the time when using $p$ cores, and $p$ is the number of cores. Basically, $E_p$ tells you the efficiency of an algorithm at $p$ cores.

Unfortunately, when doing that, absolute differences in the timings do not play a role (i.e. when one algorithm is ten times as slow as another one). To take care of that, we are using what we call the "relative parallel efficiency", which is defined by $E_p=T_{ref}/(p T_p)$, where $T_{ref}$ is the time of the fastest algorithm.

Plotting the relative parallel efficiency on a semilog scale shows pretty clearly the scaling of an algorithm, and also shows how the algorithms perform relatively to each other.

Answer 2

Georg Hager wrote about this in Fooling the Masses - Stunt 3: The log scale is your friend.

While it is true that log-log plots of strong scaling are not very discerning on the high end, they allow for showing scaling across many more orders of magnitude. To see why this is useful, consider a 3D problem with regular refinement. On a linear scale, you can reasonably show performance across about two orders of magnitude, e.g., 1024 cores, 8192 cores, and 65536 cores. It is impossible for the reader to tell from the plot whether you ran anything smaller, and realistically, the plot mostly just compares the largest two runs.

Now supposing we can fit 1 million grid cells per core in memory, this means that after strong scaling twice by a factor of 8, we can still have 16k cells per core. That is still a sizable subdomain size and we can expect many algorithms to run efficiently there. We've covered the visual spectrum of the chart (1024 to 65536 cores), but haven't even entered the regime where strong scaling becomes difficult.

Suppose instead that we started at 16 cores, also with 1 million grid cells per core. Now if we scale out to 65536 cores, we'll only have 244 cells per core, which is going to be much more discerning. A log axis is the only way to clearly represent the spectrum from 16 cores to 65536 cores. Of course you can still use a linear axis and have a caption saying "the data points for 16, 128, and 1024 cores overlap in the figure", but now you're using words instead of the figure itself to show.

A log-log scale also allows your scaling to "recover" from machine attributes like moving beyond a single node or rack. It's up to you whether this is desirable or not.

Answer 3

I agree with everything Jed had to say in his response, but I wanted add the following. I've become a fan of the way Martin Berzins and his colleagues show scaling for their Uintah framework. They plot weak and strong scaling of the code on log-log axes (using the run time per step of the method). I think it shows how the code scales pretty well (though deviation from perfect scaling is a little hard to determine). See page 7 and 8 figures 7 and 8 of this* paper for example. They also give a table with the numbers corresponding to each scaling figure.

An advantage of this is that once you've provided the numbers, there's not much a reviewer can say (or at least not much that you can't rebut).

*J. Luitjens, M. Berzins. “Improving the Performance of Uintah: A Large-Scale Adaptive Meshing Computational Framework,” In Proceedings of the 24th IEEE International Parallel and Distributed Processing Symposium (IPDPS10), Atlanta, GA, pp. 1--10. 2010. DOI: 10.1109/IPDPS.2010.5470437