These days, one can have 64 cores in a single node. I wonder how well the dense matrix-matrix product (SGEMM and DGEMM) scales with multiple CPUs/cores?

I tried to find some relevant benchmarks, but couldn't.


In comparison with things like matrix vector multiplication (in which there's no cache reuse and everything has to come out of memory), matrix-matrix multiplication allows for lots of cache reuse in a careful implementation. Performance depends on having a good implementation of BLAS and perhaps depends on how much memory bandwidth is available although is much less of an issue that it was 10-20 years ago.

Over the last decade, in my own testing, I've been seeing at least 80% parallel efficiency in DGEMM for reasonably large matrices (say N=5000) on dual socket Xeon servers with up to 8 cores running well tuned BLAS implementations (ATLAS, OpenBlas, MKL, etc.) I've never had a machine with more than 8 cores that I've tested, so I won't comment further about larger numbers of processors. Don't expect good parallel efficiency for small matrices (even N=1000 is small for this.)

  • 1
    $\begingroup$ A report from the University of Maryland discusses the scaling of dgemm performance on Maryland's "Mills" cluster: docs.hpc.udel.edu/_media/technical/whitepaper/… $\endgroup$ Jul 18 '14 at 17:26
  • $\begingroup$ Thanks for the link. I didn't even know about shared FPUs. $\endgroup$
    – bobcat
    Jul 18 '14 at 22:12
  • $\begingroup$ The shared FPU is a feature of the AMD "Bulldozer" processors. These are not processors that I would recommend for high performance scientific computing. $\endgroup$ Jul 18 '14 at 22:33
  • $\begingroup$ The basic point here is that you should be getting nearly linear speedup on matrix multiplication. $\endgroup$ Jul 18 '14 at 22:34
  • $\begingroup$ Why not? Xeon E5-4620 and Opteron 6378, for example, have comparable specs (cache and frequency), but the Opteron can use faster RAM, has 16 cores instead of 8 (but the same number of FPUs I believe) and costs half as much. $\endgroup$
    – bobcat
    Jul 18 '14 at 23:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.