19
A lot has been written about how to design, execute, and report the results of computational experiments. This has obvious connections with open source software and the broader "open science" movement. Another important issue is the difference between "my code is faster than yours" research and research that helps us to better understand the properties of ...
13
In general, both methods of performance comparisons have their place.
Comparing cpu time is in a sense the most interesting metric, because at the end of the day you are really interested in which of the methods is faster. (But make sure that the termination criteria are comparable; e.g., that both methods yield an approximation with the same accuracy). ...
12
I think it depends on your purpose. If you are trying to assess the overall performance of the code or environment, then I'd encourage you to run it however you think most people will run it on a desktop environment: leave things open but make sure nothing is crunching in the background or hogging all the memory. The biggest culprits, in my experience, are ...
10
Yes, benchmarking should be done. I make this claim as an Editor, Author, and Reviewer. Below, I represent these roles' stances slightly hyperbolically.
But let me strongman your argument first. In addition to the following claims
The benchmarking says nothing about the intrinsic speed of algorithm A versus algorithm B -- all it tells us that the ...
8
If this actually works, and it seems to, that'd be awesome because we can get a lot of undocumented data about the cache of a GPU.
A frustrating aspect of high performance computing research is digging through all the undocumented instruction sets and architecture features when trying to tune code. In HPC, the proof is in the benchmark, whether it is High ...
8
The benchmarks on the Julia website 1 2 include R and Matlab as competitors.
Note that these are benchmarks focusing on testing the pure speed of the language, not the quality of the underlying linear algebra or FFT libraries. The speed for operations that are outsourced to these libraries (such as a large matrix multiplication) can vary a lot depending on ...
7
You can calculate GFLOP rates this way, but the numbers are pretty meaningless on today's hardware:
Floating point operations require a variable number of clock cycles. An addition is generally cheaper than a multiplication, but each generally takes more than one clock cycle of the 2.8 billion cycles you quite.
When you have hyperthreading, you have two ...
6
Benchmarks are useful, but no benchmark tells the whole story. There are many useful benchmarks. For example, the Julia microbenchmarks are an interesting case of an isolating benchmark: it tries to change one thing at a time. The best example of this is the Fibonacci benchmark. The benchmark is a recursive algorithm. It's a terrible way to compute Fibonacci ...
5
In comparison with things like matrix vector multiplication (in which there's no cache reuse and everything has to come out of memory), matrix-matrix multiplication allows for lots of cache reuse in a careful implementation. Performance depends on having a good implementation of BLAS and perhaps depends on how much memory bandwidth is available although is ...
4
I don't know why Chris Rackaukas hid his answer because it's actually quite good. In the following, I'm going to assume that the program being tested is executing the same instructions every single time.
I believe that those who use averages or omit outliers in both directions have not fully understood what kind of statistics they would expect from run times....
4
In general, the answers is no.
Modern CPU's excel at problems which have high arithmetic intensity and are implemented using floating point arithmetic.
A few figures will help explain the discrepancy.
On the Haswell chip the MULPS instructions performs 8 single precision floating point multiplications. The latency is 5 CPU cycles and the reciprocal ...
4
Folks,
I found this discussion very interesting, but I was surprised to see that re-ordering the loops in the Matmul example changed the picture. I don't have an intel compiler available on my current machine, so I am using gfortran, but a rewrite of the loops in the mm_test.f90 to
call cpu_time(start)
do r=1,runs
mat_c=0.0d0
do j=1,n
...
4
I find the number of iterations to be a misleading metric because it suggests "speed" when it is not. For a simple example of comparing a few different preconditioners that shows this difference, see here: http://www.dealii.org/developer/doxygen/deal.II/step_6.html#Possibilitiesforextensions
2
In case it's not clear in the other answers, what number-of-iterations is good for is big-O arguments.
It's not good for absolute speed, because that depends on the average-time-per-iteration, which may differ between methods by a large factor.
For example, there is a tendency to ignore the cost of calculating array indices, and that may well account for a ...
2
I remember reading two or three years ago statistics about OS distribution for the List of the 500 fastest computers world wide. Linux/unix based running computer number was 498, Windows based 2 and no MacOS. Unfortunately I failed to get to the statistics again. Wikipedia's SuperComputer article just states:
"Although most modern supercomputers use the ...
2
There should be little difference in principle as the underlying tool kits are usually similar if not the same: libraries, compilers (and hardware). In practice, there can be improvement from using 'thin' nodes. That is no GUI, no email running in the back ground or any of the other many processes an OS may have going to facilitate the user experience. It ...
2
It's not quite the problem for which you're looking for solutions, but if you're willing to consider something slightly out of the box, there are numerous benchmarks for the R-T instability in the case of the Stokes equation without time derivative -- i.e., the Boussinesq approximation. Rather than list a bunch of papers, I'm simply going to reference the ...
2
EDIT
I had forgotten about this, but a while ago a friend of mine did his whole thesis work on the Rayleigh-Taylor instability in the context of statistical analysis of turbulence. You can find it enclosed here :
http://www.theses.fr/2011DENS0035
The thesis is in French, but the work is extremely thorough concerning the RT instability. You might find the ...
1
Google's brotli links to several benchmarks which look to have speed in addition to ratio:
Squash benchmark
Squash benchmark unstable
Large text compression benchmark
Lzturbo benchmark
Though you would probably need to setup benchmarks yourself
1
All benchmarks are valid in some context, the problem is that authors of benchmarks frequently do not provide the context in which to meaningfully interpret the output of their benchmarks.
The context of a benchmark is the thing it measures and all the steps taken to ensure it actually measures the thing it claims and not something else. A benchmark should ...
1
I believe this paper by Breuer et al. which uses both the lattice Boltzmann and the finite volume method could be of interest for you. There is tremendous information therein and I have used it before as a benchmark. I am not sure if it is THE reference, but it is a good one (and cited above 300 times)
http://www.sciencedirect.com/science/article/pii/...
1
There's two recent papers you should check:
GPU Performance Modeling and Optimization:
https://pure.tue.nl/ws/files/39759895/20161018_Li.pdf
Accelerating BLAS on Custom Architecture thru Algo/Arch codesign
https://arxiv.org/abs/1610.06385
1
I don't know if there is one source. Mostly just blogs and manufacturers show this kind of thing. You may want to start a blog on this! The Puget Systems HPC Blog is probably the most complete source, featuring a lot of benchmarks on Xeon processors, NVIDIA GPUs, and Xeon Phi acceleration cards. It doesn't only focus on BLAS, but the applications are all ...
1
On a side note perhaps, I think it's funny how the Ghia paper is still used as the benchmark 35 years later on. It had indeed produced great results for its time, but this being a computational problem means their accuracy was limited by the available computer power of the time and today they seem under-resolved. Actually I don't think Ghia et al. even tried ...
1
I consider this to be the "classic" 3D-Lid-Driven Cavity (LDC) incompressible flow benchmark paper:
Guj, G. & Stella, F. A vorticity-velocity method for the numerical of 3D incompressible flows. J. Comput. Phys. 298, 286–298 (1993).
Additionally, I developed a 200 line 3-D LDC incompressible flow solver in fortran:
https://github.com/charliekawczynski/...
1
I would suggest to look at the SPEC CPU2006 benchmarks, specifically the Floating Point benchmarks. They are many and complex, so you may not necessarily want to run them yourself (the also cost money) but you may be able to find results at http://www.spec.org
Only top voted, non community-wiki answers of a minimum length are eligible
