17
$\begingroup$

A lot of my own work revolves around making algorithms scale better, and one of the preferred ways of showing parallel scaling and/or parallel efficiency is to plot the performance of an algorithm/code over the number of cores, e.g.

artificial parallel scaling plot

where the $x$-axis represents the number of cores and the $y$-axis some metric, e.g. work done per unit of time. The different curves show parallel efficiencies of 20%, 40%, 60%, 80%, and 100% at 64 cores respectively.

Unfortunately though, in many publications, these results are plotted with a log-log scaling, e.g. the results in this or this paper. The problem with these log-log plots is that it's incredibly difficult to assess the actual parallel scaling/efficiency, e.g.

enter image description here

Which is the same plot as above, yet with log-log scaling. Note that now there is no big difference between the results for 60%, 80%, or 100% parallel efficiency. I've written a bit more extensively about this here.

So here's my question: What rationale is there for showing results in log-log scaling? I regularly use linear scaling to show my own results, and regularly get hammered by referees saying that my own parallel scaling/efficiency results don't look as good as the (log-log) results of others, but for the life of me I can't see why I should switch plot styles.

$\endgroup$
16
$\begingroup$

We are currently writing a paper that contains a number of comparable plots, and we more or less had the same problem. The paper is about comparing the scaling of different algorithms over the number of cores, which ranges between 1 and up to 100k on a BlueGene. The reason for using loglog-plots in this situation is the number of orders of magnitude involved. There is no way one can plot 6 orders of magnitude on a linear scale.

And indeed, when plotting the time over the number of cores in loglog, the algorithms are not very distinguishable, as you can see in the following plot. Timings of a number of algorithms on loglog-scale. The different algorithms are hard to distinguish.

Therefore, we decided to plot the parallel efficiency over the number of cores instead of the timing on a semilog-scale. The parallel efficiency is defined as $E_p=T_1/(p T_p)$, where $T_1$ is the time when using a single core, $T_p$ is the time when using $p$ cores, and $p$ is the number of cores. Basically, $E_p$ tells you the efficiency of an algorithm at $p$ cores.

Unfortunately, when doing that, absolute differences in the timings do not play a role (i.e. when one algorithm is ten times as slow as another one). To take care of that, we are using what we call the "relative parallel efficiency", which is defined by $E_p=T_{ref}/(p T_p)$, where $T_{ref}$ is the time of the fastest algorithm.

Plotting the relative parallel efficiency on a semilog scale shows pretty clearly the scaling of an algorithm, and also shows how the algorithms perform relatively to each other. Relative parallel efficiency of a number of algorithms over the number of cores.

$\endgroup$
  • 2
    $\begingroup$ I like the suggestion of plotting the parallel efficiency on a semi-log scale a lot! In my own work, I show the efficiency, relative to a reference solution, but with a linear scale along the $x$-axis. The difference in expressiveness between both your plots is quite impressive! $\endgroup$ – Pedro Mar 18 '13 at 14:15
  • $\begingroup$ Note that the plots do not look nearly as impressive as other scaling plots, as they drop off pretty quickly on the logscale. Also, you can in theory also plot the efficiency in a loglog plot to see more details at the right edge. Note however that this mean that you look in detail at very low efficiencies, which is probably not of great interest. $\endgroup$ – olenz Mar 18 '13 at 14:39
14
$\begingroup$

Georg Hager wrote about this in Fooling the Masses - Stunt 3: The log scale is your friend.

While it is true that log-log plots of strong scaling are not very discerning on the high end, they allow for showing scaling across many more orders of magnitude. To see why this is useful, consider a 3D problem with regular refinement. On a linear scale, you can reasonably show performance across about two orders of magnitude, e.g., 1024 cores, 8192 cores, and 65536 cores. It is impossible for the reader to tell from the plot whether you ran anything smaller, and realistically, the plot mostly just compares the largest two runs.

Now supposing we can fit 1 million grid cells per core in memory, this means that after strong scaling twice by a factor of 8, we can still have 16k cells per core. That is still a sizable subdomain size and we can expect many algorithms to run efficiently there. We've covered the visual spectrum of the chart (1024 to 65536 cores), but haven't even entered the regime where strong scaling becomes difficult.

Suppose instead that we started at 16 cores, also with 1 million grid cells per core. Now if we scale out to 65536 cores, we'll only have 244 cells per core, which is going to be much more discerning. A log axis is the only way to clearly represent the spectrum from 16 cores to 65536 cores. Of course you can still use a linear axis and have a caption saying "the data points for 16, 128, and 1024 cores overlap in the figure", but now you're using words instead of the figure itself to show.

A log-log scale also allows your scaling to "recover" from machine attributes like moving beyond a single node or rack. It's up to you whether this is desirable or not.

$\endgroup$
  • $\begingroup$ Thanks for the detailed reply! I hadn't read Hager. I agree that showing different scales is important, but apart from seeing them on the $x$-axis, the plot still doesn't really show anything in the $y$-axis, i.e. it is still impossible to judge the actual efficiency. In any case, I would consider scaling on a single node, a small number of nodes, and thousands of nodes to be separate problems which, in my opinion, deserve separate linear-scaled plots, each starting at a single core, node, or 1000 nodes respectively. $\endgroup$ – Pedro Mar 17 '13 at 13:53
  • 1
    $\begingroup$ It is much more difficult to strong scale a single problem by a factor of 4096 than to scale two different problem sizes by a factor of 64 each. In the example I gave, it's easy to make the two independent cases show better than 95% efficiency, but have the single combined case have less than 30% efficiency. In science and industry, there is no predetermined reason for desired turn-around time to fall within the narrow size range where the algorithm is "comfortable". $\endgroup$ – Jed Brown Mar 17 '13 at 14:10
  • $\begingroup$ I completely agree that scaling from one to thousands is the grand challenge! The reason I consider different magnitudes to be different problems is that it will mean different things for the end-user. E.g. in MD, most biologists don't have a BlueGene in the basement, but do have some multi-core workstations, or even a grant for some time on a moderate-sized cluster (small number of nodes), and people looking at large CFD problems, however, won't care much for the single-node case because the problem won't fit in memory. It's not about the algorithm's comfort, but the user's setup. $\endgroup$ – Pedro Mar 17 '13 at 14:47
2
$\begingroup$

I agree with everything Jed had to say in his response, but I wanted add the following. I've become a fan of the way Martin Berzins and his colleagues show scaling for their Uintah framework. They plot weak and strong scaling of the code on log-log axes (using the run time per step of the method). I think it shows how the code scales pretty well (though deviation from perfect scaling is a little hard to determine). See page 7 and 8 figures 7 and 8 of this* paper for example. They also give a table with the numbers corresponding to each scaling figure.

An advantage of this is that once you've provided the numbers, there's not much a reviewer can say (or at least not much that you can't rebut).

*J. Luitjens, M. Berzins. “Improving the Performance of Uintah: A Large-Scale Adaptive Meshing Computational Framework,” In Proceedings of the 24th IEEE International Parallel and Distributed Processing Symposium (IPDPS10), Atlanta, GA, pp. 1--10. 2010. DOI: 10.1109/IPDPS.2010.5470437

$\endgroup$
  • $\begingroup$ Any chance you could embed the image directly into your answer? $\endgroup$ – Aron Ahmadia Mar 17 '13 at 19:43
  • $\begingroup$ While arguably fair use to borrow their figure, I'd rather drive traffic to the authors' site. Maybe I'll make up some numbers and my own graph and come back later with a figure. $\endgroup$ – Bill Barth Mar 17 '13 at 22:57
  • $\begingroup$ From that perspective, you can wrap the image so it links to the author's site, as well as increase the amount of text in the link. If you want to discuss this more, I can open up a meta/chat thread. $\endgroup$ – Aron Ahmadia Mar 18 '13 at 9:16
  • $\begingroup$ @BillBarth Your link just redirects to their home page now. Could you fix it or embed the intended image? $\endgroup$ – Jed Brown Feb 18 '15 at 18:28
  • 1
    $\begingroup$ @JedBrown Link edited. Full reference added. DOI added. $\endgroup$ – Bill Barth Feb 18 '15 at 18:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.