# efficient MPI collective non-blocking communication

I am facing a problem with MPI (Fortran). I have a really big matrix at each node and they differ at different nodes. At some point of my calculations, each node needs the matrix from all other nodes, so that the MPI communication is between every node. Each node is a sender at the same time it is also a receiver. The matrix size is already too big to allow me to simply collect data from all other nodes. Currently what I am doing is to combine MPI_ISEND with MPI_RECV, but I think this might not be the best solution, thus, I would like to consult you experts.

To be more specific, the matrix on each node is of same size, but they are not replica of each other, their elements are different at different nodes. At some point of time, each node needs the matrix from all other nodes. Thus, each node shall send its own matrix at the same receive data from all other nodes.

I would be really grateful to you valuable suggestions and maybe solutions.

• Using Isend and Recv is typically the way to do this. Why exactly do you think this may be insufficient? Trying to optimize something that may not need to be optimized is a common mistake, and the old adage "premature optimization is the root of all evil" applies. If using Isend/Recv is not sufficient, then you will find out about this during benchmarking -- but not by just thinking that it might. – Wolfgang Bangerth Feb 8 '16 at 6:18
• You have not made it at all clear what communication pattern you are trying to implement. Code or equations would help. – Jeff Hammond Feb 9 '16 at 3:38
• I think this is not an MPI problem, but an algorithm problem. Using isend and recv is the way to go. But When to call them, how, etc. could make a difference. But you need to explain better what are doing. What algorithm are you using? what problem? what equations? some code would help too. – jbcolmenares Feb 11 '16 at 12:36
• I would focus on bypassing the "each node needs the matrix from all other nodes". This is not usually the way that things work, especially in very big problems. If you really want to do that, then maybe you should try shared-memory parallelization. If you want to investigate your communication patterns deeper, you should use a tracing tool, like Vampir. – MakisH Nov 9 '16 at 8:43
• With a lack of specifics on your particular problem, it's a bit difficult to offer suggestions. But on the off-chance that you're performing some kind of a distributed matrix-vector product A.x, you should really be distributing segments of x and operating on them with the local submatrices rather than communicating enormous arrays – sssssssssssss Dec 4 '17 at 14:57

What you're describing (each node needs every other node's data) is exactly implemented by MPI_Allgather:
You might expect MPI_Allgather to be more efficient than numerous MPI_ISEND/MPI_IRECV pairings since making MPI aware of your intended communication pattern could allow it to optimize information flow.
On the other hand, if you have wide variation in per-node calculation time, the MPI_ISEND/MPI_IRECV pairings would allow you to begin transferring information immediately as the nodes complete their operations, which might reduce latency.