I seem to lack a fundamental understanding of best practise recommendations given by Intels MKL user guides for using MKL in threaded applications. So let's clarify it together.
Wording and the question
There are especially two different ways to optimize numerical code. Either OpenMP or the Message Passing Interface (MPI) - and combinations of both. It seems to me MKL settles with OpenMP internally out of the box. Intel recommends to not combine OpenMP parallelization with a manual parallelization (e.g., a domain decomposition at higher level) with the aid of MPI (see here).
I am confused now about what to do in multi-processor environments (read clusters) with many physical computation units. My first approach towards the topic was:
- Use MPI, do a domain decomposition of the problem, distribute the small chunks to all CPUs in the cluster and use sequential MKL (=
MKL without OpenMP) there.
I especially ask myself how it could be done differently when relying only on multi-threaded MKL (=
MKL with OpenMP) in the cluster: What if one matrix in a matrix vector product is to large to fit into memory of one compute node? Will it be automatically spread over all compute nodes by MKL?