I've seen a lot of publications in Computational Physics journals use different metrics for the performance of their code. Especially for GPGPU code, there seems to be a great variety of timing results people publish. In particular, I've seen:
- Comparisons of (essentially) running
timeon the GPU and CPU versions and reporting an average
- Comparisons of profiling data on each GPU/CPU function called (so timing the run of
mainand all functions called from it, but ignoring driver initialization time)
- Comparisons of profiling data for several relevant functions, ignoring things like memory transfers (across the PCI-E bus in this case), I/O to the disk, transforming the data from one format to another, etc
- Comparisons of profiling data for only one function (for instance, comparing only the time to do spin updates in a CPU vs GPU Quantum Monte Carlo) and ignoring things like memory transfer time, I/O to disk, setting up the Hamiltonian/diagonalizing it, etc
My feeling is that the first option is the most 'honest' of the four, although I can see the merits of the second and third as well. It's a bit ridiculous to report a runtime difference of 5s when 4.99s of it was the difference in I/O implementations between languages or Infiniband vs Gigabit. The last option seems a bit "sketchy" to me, since unless this function is the pain point of the whole program reporting information about it only won't reflect the performance someone replicating my results would see. Which of these approaches is more forthright? Ideally a paper would contain all of these but in the case of a limitation on figures/length, which is/are most valuable, honest, and relevant to provide?