I'm compiling a blas to use with scalapack. Some blases, such as openblas, are naturally multithreaded, but scalapack uses mpi, so maybe needs a single-threaded blas? My questions then are:

• does scalapack need a multi-threaded blas? a single-threaded blas? or it will work just fine on either because, for example, it's somehow forcing the number of threads to 1 at runtime?
• in the case that scalapack forces the number of threads to 1 at runtime, how does it do this? does this work with openblas? It seems that the api for this in openblas is openblas_set_num_threads(1), which doesn't sound like a particularly portable command, and sounds like such configuration is blas-specific?

Edit: I suppose I could just use a multithreaded OpenBLAS, and call openblas_set_num_threads(1) myself?

You have to choose. It may be that running a threaded BLAS with a reduced MPI rank count gives better performance than running lots of ranks with one thread per rank. If you have $p$ processor cores, then any pair of integers $t$ and $r$ with $tr \le p$ is a viable possibility for combinations of threads ($t$) and ranks ($r$). The fastest of which will depend on your machine and your problem. Typical choices would be $r=p$ with $t=1$ and $r=s$ with $t=p/s$ where $s$ is the number of CPU sockets in the node. At the very least you should test these two. Don't forget to choose an appropriate binding of ranks and threads to sockets, or your performance may suffer.
• Thanks. And then, we manually set the number of threads directly on the blas, e.g. using openblas_set_num_threads(1)? Or scalapack handles this for us? – Hugh Perkins Jun 18 '13 at 23:52
• You usually need to set it manually, though I would have said openblas_set_num_threads(t) to remain consistent with my post. Depending on your BLAS, you'll have to set that using the parameters it requires. The Intel MKL (which I know the most about) conveniently reads the environment variables OMP_NUM_THREADS or MKL_NUM_THREADS, so you can set the value at run time easily. I recommend explicitly setting it yourself so that any defaults the BLAS has are overridden. – Bill Barth Jun 19 '13 at 9:00