# How does matrix-matrix product scale with multiple CPUs?

These days, one can have 64 cores in a single node. I wonder how well the dense matrix-matrix product (SGEMM and DGEMM) scales with multiple CPUs/cores?

I tried to find some relevant benchmarks, but couldn't.

## 1 Answer

In comparison with things like matrix vector multiplication (in which there's no cache reuse and everything has to come out of memory), matrix-matrix multiplication allows for lots of cache reuse in a careful implementation. Performance depends on having a good implementation of BLAS and perhaps depends on how much memory bandwidth is available although is much less of an issue that it was 10-20 years ago.

Over the last decade, in my own testing, I've been seeing at least 80% parallel efficiency in DGEMM for reasonably large matrices (say N=5000) on dual socket Xeon servers with up to 8 cores running well tuned BLAS implementations (ATLAS, OpenBlas, MKL, etc.) I've never had a machine with more than 8 cores that I've tested, so I won't comment further about larger numbers of processors. Don't expect good parallel efficiency for small matrices (even N=1000 is small for this.)

• A report from the University of Maryland discusses the scaling of dgemm performance on Maryland's "Mills" cluster: docs.hpc.udel.edu/_media/technical/whitepaper/… – Brian Borchers Jul 18 '14 at 17:26
• Thanks for the link. I didn't even know about shared FPUs. – MaxB Jul 18 '14 at 22:12
• The shared FPU is a feature of the AMD "Bulldozer" processors. These are not processors that I would recommend for high performance scientific computing. – Brian Borchers Jul 18 '14 at 22:33
• The basic point here is that you should be getting nearly linear speedup on matrix multiplication. – Brian Borchers Jul 18 '14 at 22:34
• Why not? Xeon E5-4620 and Opteron 6378, for example, have comparable specs (cache and frequency), but the Opteron can use faster RAM, has 16 cores instead of 8 (but the same number of FPUs I believe) and costs half as much. – MaxB Jul 18 '14 at 23:44