8
$\begingroup$

I've seen a lot of publications in Computational Physics journals use different metrics for the performance of their code. Especially for GPGPU code, there seems to be a great variety of timing results people publish. In particular, I've seen:

  • Comparisons of (essentially) running time on the GPU and CPU versions and reporting an average
  • Comparisons of profiling data on each GPU/CPU function called (so timing the run of main and all functions called from it, but ignoring driver initialization time)
  • Comparisons of profiling data for several relevant functions, ignoring things like memory transfers (across the PCI-E bus in this case), I/O to the disk, transforming the data from one format to another, etc
  • Comparisons of profiling data for only one function (for instance, comparing only the time to do spin updates in a CPU vs GPU Quantum Monte Carlo) and ignoring things like memory transfer time, I/O to disk, setting up the Hamiltonian/diagonalizing it, etc

My feeling is that the first option is the most 'honest' of the four, although I can see the merits of the second and third as well. It's a bit ridiculous to report a runtime difference of 5s when 4.99s of it was the difference in I/O implementations between languages or Infiniband vs Gigabit. The last option seems a bit "sketchy" to me, since unless this function is the pain point of the whole program reporting information about it only won't reflect the performance someone replicating my results would see. Which of these approaches is more forthright? Ideally a paper would contain all of these but in the case of a limitation on figures/length, which is/are most valuable, honest, and relevant to provide?

|cite|improve this question
$\endgroup$
6
$\begingroup$

Total running time (wall clock) is the only metric that matters in industry or real life applications: this figure should never be omitted, even if embarrassing. Of course this metric is very dependent on the test environment, so this should be described in detail.

All other metrics can (or should be) reported if they provide insight into the problem, or interesting conclusions (e.g. verification of some theoretical bound) can be drawn from them.

I think that this blog entry addresses this point, especially the last paragraph.

|cite|improve this answer
$\endgroup$
  • 2
    $\begingroup$ Even wall clock time strikes me as possibly misleading, because the quality of the implementations of each algorithm matters. Joel Spolsky addresses the issue of "no such thing as the fastest code" in a blog post. Without posting the source code of the implementations (and I suspect in some cases, even if one did post the source code of the implementations), how am I to know what's really "fastest"? Put another way, how does one make a comparison of wall clock times (or any other metric) meaningful? $\endgroup$ – Geoff Oxberry Jul 19 '12 at 10:12
  • 1
    $\begingroup$ @GeoffOxberry In a distant past almost every computation was serial, and the memory hierarchy was just two levels, in-core and out-of-core. In those (happy) days matlab still reported flops count after each command... Nowadays we have multicore CPUS, GPGPUS, clusters, clouds, L1/L2/L3 caches, ... Efficiency is determined by how well you are able to map an algorithm to the given hw/sw architecture. It is silly to try to condense everything to a single figure, but nevertheless we should be able to introduce an ordering, and tell, at some given and well defined conditions, who is faster. $\endgroup$ – Stefano M Jul 19 '12 at 11:03
  • $\begingroup$ Yes, I agree. I'm asking how one should introduce such an ordering in order to make a meaningful comparisons that will determine, under whatever given and well-defined conditions you specify, who is faster. $\endgroup$ – Geoff Oxberry Jul 19 '12 at 21:01
  • $\begingroup$ +1 All this makes sense, and it depends on the point one is trying to make in the publication. If the point is about performance, then it needs to be explained a lot more carefully. $\endgroup$ – Mike Dunlavey Jul 20 '12 at 13:28
4
$\begingroup$

It is often the case that one can only report the tip of the iceberg of all the work and compromises that went into a piece of software. Reporting performance is nice but the real deal is when the code is made freely accesible on internet, this way, anyone interested can evaluate and reproduce the results.

Ideally, if you release the software, you can also make available the tests that generate the data presented in a paper.

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ I completely agree: bare figures make no sense. In order to have reproducible and meaningful results source code, test data sets, hw configuration, sw configuration, should be made public. $\endgroup$ – Stefano M Jul 19 '12 at 11:06
  • $\begingroup$ In the case that I had implemented an algorithm twice, once on the GPU and once on the CPU, I agree that HW/SW configurations are absolutely necessary to include. I'm still unclear on which data to present in the case that I find a "scaling effect" (for instance, comparing speedup vs matrix size for ED). Should I compare wallclocks? Just the ED part of the calculation? $\endgroup$ – limes Jul 20 '12 at 19:11
  • $\begingroup$ If you are interested on a deep comparison of codes, I like the work of Volkov and Demmel, in their (a bit old but illustrative) paper: Benchmarking GPUs to tune dense linear algebra $\endgroup$ – fcruz Jul 22 '12 at 9:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.