6
$\begingroup$

I'm not sure if I did something wrong or if I just didn't understand the concept of an optimized BLAS.

I'm a FEM engineer trying to optimize my setup on a small cluster computer (six nodes). I'm building everything on open source tools. My aim is to use a robust setup which allows me to manage large mechanical simulations. After finally compiling my programs using MPI I'm reading the using an optimized BLAS will help me further pushing down calculation time.

I installed OpenBlas and built Lapack 3.4.2. The results of the lapack benchmarks were surprising. They are fastest with the system (unoptimzed) BLAS and the run time increases with increasing number of threads specified (export OPENBLAS_NUM_THREADS=i). The same is true for scalapack. Furthermore, i>16 doesn't seem to utilize more than 16 threads, although I have 32 available on this machine. That's ok, because I want some to run a couple of MPI processes (4?) on one node anyways, but I find it suspicious.

I'm a beginner and don't know where to start digging from here. Can someone give me a hint? I have a pastebin of the i=32 log here.

OpenBLAS build complete.
OS               ... Linux
Architecture     ... x86_64 
BINARY           ... 64bit
C compiler       ... GCC  (command line : gcc)
Fortran compiler ... GFORTRAN  (command line : gfortran)
Library Name     ... libopenblas_sandybridgep-r0.2.5.a (Multi threaded; Max num-threads is 32)

Lapack 3.4.2 make.inc (changes from default)

 BLASLIB = /path/to/libopenblas.a -lpthread

-- EDIT: (more information because of @Aron Ahmadia's comments)

  • system BLAS log
  • i=1 log
  • i=32 log
  • CentOS release 6.3 (Final)
  • blas-devel-3.2.1-4.el6.x86_64
  • gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) (same for gfortran).

  • MemTotal: 198202172 kB

    $ grep "Xeon(R) CPU E5-2690 0 @ 2.90GHz" /proc/cpuinfo | wc -l 
32.
| cite | improve this question | |
$\endgroup$
  • $\begingroup$ What Linux distribution are you running (release number, etc...), what CPUs are on your system, and what version of the compilers are you using? The system BLAS on many distributions is not necessarily a poorly tuned one, but I agree that your results are surprising. $\endgroup$ – Aron Ahmadia Jan 29 '13 at 23:16
  • 1
    $\begingroup$ I'm puzzled. The tuned OpenBLAS should be much faster than the RPM package you have installed, particularly on a single core. Can you provide the benchmarks that you ran as well as their timing? Was the machine otherwise idle when you ran the tests? $\endgroup$ – Aron Ahmadia Jan 29 '13 at 23:28
  • $\begingroup$ I'll need to do the timing. What time is desired, bash time or /usr/bin/time? machines are idle. This is the log for the system BLAS. $\endgroup$ – Sebastian Jan 29 '13 at 23:31
  • 1
    $\begingroup$ Ahh, it looks like you are running verification tests as opposed to performance benchmarks. I'm guessing that the wall time here is mostly coming from the time it takes to load each program and execute it, as opposed to the actual time spent in a single problem. I actually have not benchmarked BLAS/LAPACK in awhile, but you are looking to run a larger problem that should take your CPU a minute or so to solve. $\endgroup$ – Aron Ahmadia Jan 29 '13 at 23:51
  • $\begingroup$ Okay, makes sense. Perhaps something from here? netlib.org/benchmark $\endgroup$ – Sebastian Jan 29 '13 at 23:57
5
$\begingroup$

OpenBLAS is designed as a library to perform effectively for the most common computationally challenging problems that arise in scientific computing, linear algebra problems that take minutes or longer to solve. As @Stefano and @JedBrown hint in the comments, you are not necessarily likely to depend heavily on the performance of your BLAS library for the performance of your FEM solver, and it is more important to use a freely available robust software framework such as PETSc, (potentially with deal.II or FEniCS as a friendlier FEM interface above), to develop your software. I can think of very few situations where I would not recommend using a high-level FEM and linear algebra framework to develop and execute your code.

In this case, you are not seeing a difference in performance because you are using the validation tests for LAPACK, which solves considerably smaller problems. The run time of launching the processes from the shell and loading them dynamically are confounding your performance measurements.

As suggested in the comments, you will notice a sharp relative improvement in performance by measuring larger problems.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.