# CPU benchmarks for numerical kernels

CPU benchmarks available online mostly focus on desktop apps/games and rarely on serial/parallel numerical kernels, specially sparse ones (e.g., MatMult). Some benchmarks like NAS/SciMark exists but are rarely used by popular benchmarking sites on lastest/greatest CPUs.

When it comes to codes that use sparse/dense numerical libraries such as PETSc/ScaLAPACK how does one decide which machine to buy/use specially w.r.t. to multicore performance? E.g. would I be better off with an AMD FX8350 or a Intel Core i7-3770K as my Desktop or should I request allocation on a Xeon or Opteron based cluster, specially for small/medium (i.e., less than 4000 cores) type jobs where one has more options.

I know from personal experience that Intel chips have performed significantly better in the last few years but what about the latest AMD stuff?

EDIT: I am specifically interested in PETSc's MatMult kernel (for unstructured FE matrices) which I have been told is memory bound. Performance in terms of percentage of peak FLOPS using all cores I think would be useful.

-
please expand a little more about exactly what codes you intend to be running and some more of their performance characteristics. For example, there's a huge difference between the type of machine I'd recommend for somebody with a non-scalable, memory-bound sparse PDE code that hasn't been ported beyond x86 vs somebody with a scalable dense problem that has been ported to CUDA+MPI and is floating-point bound. –  Aron Ahmadia Nov 1 '12 at 16:26
Yes it is a sparse PDE based code that can use both iterative and sparse direct solvers (e.g., MUMPS) via PETSc and I am interested in AMD's latest offerings (Opteron 62XX and FX8350 on the Desktop). –  stali Nov 1 '12 at 16:38
please update your question, title and tags. I'd do it for you but I know you know how to edit questions :) –  Aron Ahmadia Nov 1 '12 at 17:39
Specifically, I know the PETSc developers follow the petsc tag, so it's helpful to put it on questions that relate to PETSc (and they know quite a bit about the performance of various architectures). The final thing to note is what the fastest known solvers are for your problem (direct/multigrid etc.) and if you are targeting desktop or cluster performance (and if a cluster/supercomputer, at what size problem). –  Aron Ahmadia Nov 1 '12 at 17:41
Have you looked at spec.org/cpu2006 ? They have floating point and integer benchmarks for simulation codes on tons of different hardware. I don't know if any of the benchmarks use PETSc, but you may find one that does similar work to what you are considering. –  tpg2114 Nov 1 '12 at 18:19

First, sparse direct is completely different from sparse iterative. You cannot reliably predict performance if you don't have a good understanding of what your code is doing. For sparse MatMult, MatSolve, MatSOR, and similar kernels, you have an arithmetic intensity of no more than 1 flop/4 bytes of memory bandwidth. Meanwhile, most recent multicore chips can perform about 5 flops/byte. This means that even if cache is reused perfectly, you cannot get better than $1/20 = 5\%$ of floating point peak. This is a hard barrier that no amount of "tuning" can circumvent. In practice, it is very important to order your matrices so that cache is reused well. If you choose an ordering that produces poor reuse, the realized peak will be several times lower. In practice, with decent orderings and sufficiently large subdomains, PETSc sparse matrix kernels frequently get $80\%$ of memory bandwidth peak or better. Look at the STREAM benchmarks to determine the realizable bandwidth. Some machines, such as Ranger at TACC, cannot realize better than $50\%$ of peak bandwidth.

If you resort to assembled sparse matrices to be solved using iterative methods, there is absolutely no point buying cores. Buy memory bandwidth. I recommend testing algorithms with assembled matrices, then switching to mostly-unassembled methods (perhaps matrix-free multigrid), when appropriate, if you want to get high machine utilization. A better choice of algorithm can give an order of magnitude speedup in so many cases that there's no point micro-tuning a suboptimal method.

-
This is very helpful because the STREAM benchmarks are fairly common. –  stali Nov 2 '12 at 1:58

There are benchmarks that are derived from typical scientific computing codes doing sparse matrix vector stuff in SPEC CPU 2006, see http://www.spec.org/cpu2006/ . In particular, the 447.dealII benchmark in this suite derives from the deal.II library and spends most of its time in exactly the kind of operations that you describe for your workload. The advantage of this is that you can query the SPEC database for the peak rates on this benchmark to see how various processors (variants) perform on this one benchmark. Pretty much every CPU built over the past 10 years is in that database.

(Disclaimer: I'm the author of 447.dealII.)

-