CPU benchmarks for numerical kernels

Question

CPU benchmarks available online mostly focus on desktop apps/games and rarely on serial/parallel numerical kernels, specially sparse ones (e.g., MatMult). Some benchmarks like NAS/SciMark exists but are rarely used by popular benchmarking sites on lastest/greatest CPUs.

When it comes to codes that use sparse/dense numerical libraries such as PETSc/ScaLAPACK how does one decide which machine to buy/use specially w.r.t. to multicore performance? E.g. would I be better off with an AMD FX8350 or a Intel Core i7-3770K as my Desktop or should I request allocation on a Xeon or Opteron based cluster, specially for small/medium (i.e., less than 4000 cores) type jobs where one has more options.

I know from personal experience that Intel chips have performed significantly better in the last few years but what about the latest AMD stuff?

EDIT: I am specifically interested in PETSc's MatMult kernel (for unstructured FE matrices) which I have been told is memory bound. Performance in terms of percentage of peak FLOPS using all cores I think would be useful.

Answer 1

First, sparse direct is completely different from sparse iterative. You cannot reliably predict performance if you don't have a good understanding of what your code is doing. For sparse MatMult, MatSolve, MatSOR, and similar kernels, you have an arithmetic intensity of no more than 1 flop/4 bytes of memory bandwidth. Meanwhile, most recent multicore chips can perform about 5 flops/byte. This means that even if cache is reused perfectly, you cannot get better than $1/20 = 5\%$ of floating point peak. This is a hard barrier that no amount of "tuning" can circumvent. In practice, it is very important to order your matrices so that cache is reused well. If you choose an ordering that produces poor reuse, the realized peak will be several times lower. In practice, with decent orderings and sufficiently large subdomains, PETSc sparse matrix kernels frequently get $80\%$ of memory bandwidth peak or better. Look at the STREAM benchmarks to determine the realizable bandwidth. Some machines, such as Ranger at TACC, cannot realize better than $50\%$ of peak bandwidth.

If you resort to assembled sparse matrices to be solved using iterative methods, there is absolutely no point buying cores. Buy memory bandwidth. I recommend testing algorithms with assembled matrices, then switching to mostly-unassembled methods (perhaps matrix-free multigrid), when appropriate, if you want to get high machine utilization. A better choice of algorithm can give an order of magnitude speedup in so many cases that there's no point micro-tuning a suboptimal method.

Answer 2

There are benchmarks that are derived from typical scientific computing codes doing sparse matrix vector stuff in SPEC CPU 2006, see http://www.spec.org/cpu2006/ . In particular, the 447.dealII benchmark in this suite derives from the deal.II library and spends most of its time in exactly the kind of operations that you describe for your workload. The advantage of this is that you can query the SPEC database for the peak rates on this benchmark to see how various processors (variants) perform on this one benchmark. Pretty much every CPU built over the past 10 years is in that database.

(Disclaimer: I'm the author of 447.dealII.)

Answer 3

Raw benchmarks are correlated with performance for certain, but to have a real answer about what is optimal for your particular problem, you need to profile your code on the system in question. IMHO it is impossible to say authoritatively without some actual performance testing on your code.

That said, as mentioned in the question thread, most matrix/vector operations are memory bound, and become fairly dependent on so many factors, like structure of the matrix, how your compiler optimizes memory access, cache-hits, ram latency, OS foolery, the list goes on and on. I think it is very hard to make any substantive claims without actual testing on the system in question.

I have had similar experience with Intel's hardware as you mention. Especially when using the icc tool-chain, as it seems to really know how to write machine code that uses the full capabilities of the underlying metal.