# Threaded OpenBlas benchmark slower than system blas?

I'm not sure if I did something wrong or if I just didn't understand the concept of an optimized BLAS.

I'm a FEM engineer trying to optimize my setup on a small cluster computer (six nodes). I'm building everything on open source tools. My aim is to use a robust setup which allows me to manage large mechanical simulations. After finally compiling my programs using MPI I'm reading the using an optimized BLAS will help me further pushing down calculation time.

I installed OpenBlas and built Lapack 3.4.2. The results of the lapack benchmarks were surprising. They are fastest with the system (unoptimzed) BLAS and the run time increases with increasing number of threads specified (export OPENBLAS_NUM_THREADS=i). The same is true for scalapack. Furthermore, i>16 doesn't seem to utilize more than 16 threads, although I have 32 available on this machine. That's ok, because I want some to run a couple of MPI processes (4?) on one node anyways, but I find it suspicious.

I'm a beginner and don't know where to start digging from here. Can someone give me a hint? I have a pastebin of the i=32 log here.

OpenBLAS build complete.
OS               ... Linux
Architecture     ... x86_64
BINARY           ... 64bit
C compiler       ... GCC  (command line : gcc)
Fortran compiler ... GFORTRAN  (command line : gfortran)


Lapack 3.4.2 make.inc (changes from default)

 BLASLIB = /path/to/libopenblas.a -lpthread


• system BLAS log
• i=1 log
• i=32 log
• CentOS release 6.3 (Final)
• blas-devel-3.2.1-4.el6.x86_64
• gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) (same for gfortran).
• MemTotal: 198202172 kB

\$ grep "Xeon(R) CPU E5-2690 0 @ 2.90GHz" /proc/cpuinfo | wc -l
32.

• What Linux distribution are you running (release number, etc...), what CPUs are on your system, and what version of the compilers are you using? The system BLAS on many distributions is not necessarily a poorly tuned one, but I agree that your results are surprising. – Aron Ahmadia Jan 29 '13 at 23:16
• I'm puzzled. The tuned OpenBLAS should be much faster than the RPM package you have installed, particularly on a single core. Can you provide the benchmarks that you ran as well as their timing? Was the machine otherwise idle when you ran the tests? – Aron Ahmadia Jan 29 '13 at 23:28
• I'll need to do the timing. What time is desired, bash time or /usr/bin/time? machines are idle. This is the log for the system BLAS. – Sebastian Jan 29 '13 at 23:31
• Ahh, it looks like you are running verification tests as opposed to performance benchmarks. I'm guessing that the wall time here is mostly coming from the time it takes to load each program and execute it, as opposed to the actual time spent in a single problem. I actually have not benchmarked BLAS/LAPACK in awhile, but you are looking to run a larger problem that should take your CPU a minute or so to solve. – Aron Ahmadia Jan 29 '13 at 23:51
• Okay, makes sense. Perhaps something from here? netlib.org/benchmark – Sebastian Jan 29 '13 at 23:57