These days, one can have 64 cores in a single node. I wonder how well the dense matrix-matrix product (
DGEMM) scales with multiple CPUs/cores?
I tried to find some relevant benchmarks, but couldn't.
In comparison with things like matrix vector multiplication (in which there's no cache reuse and everything has to come out of memory), matrix-matrix multiplication allows for lots of cache reuse in a careful implementation. Performance depends on having a good implementation of BLAS and perhaps depends on how much memory bandwidth is available although is much less of an issue that it was 10-20 years ago.
Over the last decade, in my own testing, I've been seeing at least 80% parallel efficiency in DGEMM for reasonably large matrices (say N=5000) on dual socket Xeon servers with up to 8 cores running well tuned BLAS implementations (ATLAS, OpenBlas, MKL, etc.) I've never had a machine with more than 8 cores that I've tested, so I won't comment further about larger numbers of processors. Don't expect good parallel efficiency for small matrices (even N=1000 is small for this.)