Take the 2-minute tour ×
Computational Science Stack Exchange is a question and answer site for scientists using computers to solve scientific problems. It's 100% free, no registration required.

CPU benchmarks available online mostly focus on desktop apps/games and rarely on serial/parallel numerical kernels, specially sparse ones (e.g., MatMult). Some benchmarks like NAS/SciMark exists but are rarely used by popular benchmarking sites on lastest/greatest CPUs.

When it comes to codes that use sparse/dense numerical libraries such as PETSc/ScaLAPACK how does one decide which machine to buy/use specially w.r.t. to multicore performance? E.g. would I be better off with an AMD FX8350 or a Intel Core i7-3770K as my Desktop or should I request allocation on a Xeon or Opteron based cluster, specially for small/medium (i.e., less than 4000 cores) type jobs where one has more options.

I know from personal experience that Intel chips have performed significantly better in the last few years but what about the latest AMD stuff?

EDIT: I am specifically interested in PETSc's MatMult kernel (for unstructured FE matrices) which I have been told is memory bound. Performance in terms of percentage of peak FLOPS using all cores I think would be useful.

share|improve this question
    
please expand a little more about exactly what codes you intend to be running and some more of their performance characteristics. For example, there's a huge difference between the type of machine I'd recommend for somebody with a non-scalable, memory-bound sparse PDE code that hasn't been ported beyond x86 vs somebody with a scalable dense problem that has been ported to CUDA+MPI and is floating-point bound. –  Aron Ahmadia Nov 1 '12 at 16:26
    
Yes it is a sparse PDE based code that can use both iterative and sparse direct solvers (e.g., MUMPS) via PETSc and I am interested in AMD's latest offerings (Opteron 62XX and FX8350 on the Desktop). –  stali Nov 1 '12 at 16:38
    
please update your question, title and tags. I'd do it for you but I know you know how to edit questions :) –  Aron Ahmadia Nov 1 '12 at 17:39
    
Specifically, I know the PETSc developers follow the petsc tag, so it's helpful to put it on questions that relate to PETSc (and they know quite a bit about the performance of various architectures). The final thing to note is what the fastest known solvers are for your problem (direct/multigrid etc.) and if you are targeting desktop or cluster performance (and if a cluster/supercomputer, at what size problem). –  Aron Ahmadia Nov 1 '12 at 17:41
    
Have you looked at spec.org/cpu2006 ? They have floating point and integer benchmarks for simulation codes on tons of different hardware. I don't know if any of the benchmarks use PETSc, but you may find one that does similar work to what you are considering. –  tpg2114 Nov 1 '12 at 18:19
show 3 more comments

3 Answers

First, sparse direct is completely different from sparse iterative. You cannot reliably predict performance if you don't have a good understanding of what your code is doing. For sparse MatMult, MatSolve, MatSOR, and similar kernels, you have an arithmetic intensity of no more than 1 flop/4 bytes of memory bandwidth. Meanwhile, most recent multicore chips can perform about 5 flops/byte. This means that even if cache is reused perfectly, you cannot get better than $1/20 = 5\%$ of floating point peak. This is a hard barrier that no amount of "tuning" can circumvent. In practice, it is very important to order your matrices so that cache is reused well. If you choose an ordering that produces poor reuse, the realized peak will be several times lower. In practice, with decent orderings and sufficiently large subdomains, PETSc sparse matrix kernels frequently get $80\%$ of memory bandwidth peak or better. Look at the STREAM benchmarks to determine the realizable bandwidth. Some machines, such as Ranger at TACC, cannot realize better than $50\%$ of peak bandwidth.

If you resort to assembled sparse matrices to be solved using iterative methods, there is absolutely no point buying cores. Buy memory bandwidth. I recommend testing algorithms with assembled matrices, then switching to mostly-unassembled methods (perhaps matrix-free multigrid), when appropriate, if you want to get high machine utilization. A better choice of algorithm can give an order of magnitude speedup in so many cases that there's no point micro-tuning a suboptimal method.

share|improve this answer
    
This is very helpful because the STREAM benchmarks are fairly common. –  stali Nov 2 '12 at 1:58
add comment

There are benchmarks that are derived from typical scientific computing codes doing sparse matrix vector stuff in SPEC CPU 2006, see http://www.spec.org/cpu2006/ . In particular, the 447.dealII benchmark in this suite derives from the deal.II library and spends most of its time in exactly the kind of operations that you describe for your workload. The advantage of this is that you can query the SPEC database for the peak rates on this benchmark to see how various processors (variants) perform on this one benchmark. Pretty much every CPU built over the past 10 years is in that database.

(Disclaimer: I'm the author of 447.dealII.)

share|improve this answer
add comment

Raw benchmarks are correlated with performance for certain, but to have a real answer about what is optimal for your particular problem, you need to profile your code on the system in question. IMHO it is impossible to say authoritatively without some actual performance testing on your code.

That said, as mentioned in the question thread, most matrix/vector operations are memory bound, and become fairly dependent on so many factors, like structure of the matrix, how your compiler optimizes memory access, cache-hits, ram latency, OS foolery, the list goes on and on. I think it is very hard to make any substantive claims without actual testing on the system in question.

I have had similar experience with Intel's hardware as you mention. Especially when using the icc tool-chain, as it seems to really know how to write machine code that uses the full capabilities of the underlying metal.

share|improve this answer
    
I would love to SSH into a machine but the question is where to find one with the latest/greatest CPU. I think most users dont have access to the newest stuff. Lets say you're requesting CPU time on a cluster and you need to choose one that uses Xeon vs one that uses Opteron or maybe you're buying a Desktop. Now what will you do. Though I do understand that benchmarking is not an easy task given all the factors that you mention. –  stali Nov 1 '12 at 20:41
    
Also as far as sparse MatMult goes I havent seen a difference of more than 35% (on matrices arising from unstructured low order FE code) across GNU, Intel, Open64, Cray, PGI and Solaris Studio compilers (even with different opt switches). However I have seen a Xeon that is about twice as fast over older Opterons. Though I am not sure about the new ones which really led to the question. –  stali Nov 1 '12 at 20:52
    
Most computing centers that allow you to apply for time allow you to apply for a "startup" or "benchmarking" allocation in order to see how your code performs. We do this so that folks can write the best-informed proposals possible. –  Bill Barth Nov 1 '12 at 22:50
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.