I am reposting here a question previously asked in stackoverflow: as suggested in a reply to my question, scicomp could be a better place to obtain some useful comments/suggestions for my problem.
I am trying to make some tests with BLACS/Scalapack (C interface, Intel MKL version) making use multiple MPI communicators: particularly, what I am trying to obtain is a set of BLACS contexts/grids corresponding (one-to-one) to a set of (disjoint) MPI communicators. While I have no problems in working with a single "global" communicator (
MPI_COMM_WORLD) I have some difficulties with the multiple communicator case. I hope that you could give me some suggestions for the following problem.
I am a bit confused about the BLACS context variable value update after a call to
Cblacs_gridinit: let's suppose to start with a "global" context corresponding to the
MPI_COMM_WORLD communicator. I can obtain the corresponding BLACS context with the call:
MPI_Comm globalCommunicator(MPI_COMM_WORLD); MKL_INT globalContext(Csys2blacs_handle(globalCommunicator));
and create a grid on it with
Cblacs_gridinit(&globalContext, &c_blacsGridOrdering, MKL_INT i_nTaskRow, MKL_INT i_nTaskCol);
with, for example,
The globalContext value in this case is 0.
At some point in my code, which is making use of 6 MPI tasks,
an MPI communicator (
localCommunicator) corresponding to the group of 4 tasks with id
[0;3] is being created: at this point I would like to create a new BLACS context (
localContext) for this "local" communicator and a local grid on it. I can do that with the code
MKL_INT localContext(Csys2blacs_handle(localCommunicator)); Cblacs_gridinit(&localContext, &c_blacsGridOrdering, MKL_INT i_nTaskRowLocal, MKL_INT i_nTaskColLocal);
where the above function call sequence is performed only by the tasks included in the local communicator.
localContext value, after the call to
Csys2blacs_handle, is equal to 1 (for each task of the local communicator) but it is modified and set to 0 by the subsequent call to
Obviously this causes some problems in the rest of my code since, for example, if I try to retrieve the MPI communicator corresponding to
localContext with a call to
Cblacs2sys_handle I get a six task communicator corresponding to the initial
Most likely I am making some very stupid error (missing call to some BLACS functions??) or it is not clear to me the interplay between multiple MPI communicators and BLACS context/grids but I can not find what is wrong with my code.
Do you have some suggestions concerning the above problem? Many thanks for your help!
///////// UPDATE 1 /////////
I have an update for my question which could be useful to find a solution or at least an explanation for the observed problem: the described problem is no more present if I initialize the first grid (the one making use of the global context) on a task grid whose size is such that all available MPI tasks are included, for example with MKL_INT i_nTaskRow(1) and MKL_INT i_nTaskCol(MPI_COMM_WORLD size). Is this behavior expected from BLACS? Many thanks again for your support!