# General understanding of Intel MKL, threads and MPI

Preface

I seem to lack a fundamental understanding of best practise recommendations given by Intels MKL user guides for using MKL in threaded applications. So let's clarify it together.

Wording and the question

There are especially two different ways to optimize numerical code. Either OpenMP or the Message Passing Interface (MPI) - and combinations of both. It seems to me MKL settles with OpenMP internally out of the box. Intel recommends to not combine OpenMP parallelization with a manual parallelization (e.g., a domain decomposition at higher level) with the aid of MPI (see here).

I am confused now about what to do in multi-processor environments (read clusters) with many physical computation units. My first approach towards the topic was:

• Use MPI, do a domain decomposition of the problem, distribute the small chunks to all CPUs in the cluster and use sequential MKL (=MKL without OpenMP) there.

I especially ask myself how it could be done differently when relying only on multi-threaded MKL (=MKL with OpenMP) in the cluster: What if one matrix in a matrix vector product is to large to fit into memory of one compute node? Will it be automatically spread over all compute nodes by MKL?

• I fail to see in your link to the Intel website where it says anything that implies that you should not combine OpenMP with MPI. Could you specify what passage on that page you're referring to? – LedHead Feb 18 '18 at 17:55
• @led23head Maybe I got Intel wrong but the sentence Your application is intended to be run on a single thread, like a message-passing Interface (MPI) application seems to suggest that nested parallelism by hybrid MPI+OpenMP is not recommended. Might be wrong, though. – pbx Feb 18 '18 at 22:04
• It's saying that you should almost always use the threaded library. There are some special cases in which you might want to use the sequential library, and one of them might be if your code is MPI parallelized. I think it's talking about the case in which you plan to have multiple MPI processes running on each compute node. It's not saying you shouldn't do OpenMP inside MPI. – LedHead Feb 19 '18 at 0:30