0
$\begingroup$

I am facing a problem with MPI (Fortran). I have a really big matrix at each node and they differ at different nodes. At some point of my calculations, each node needs the matrix from all other nodes, so that the MPI communication is between every node. Each node is a sender at the same time it is also a receiver. The matrix size is already too big to allow me to simply collect data from all other nodes. Currently what I am doing is to combine MPI_ISEND with MPI_RECV, but I think this might not be the best solution, thus, I would like to consult you experts.

To be more specific, the matrix on each node is of same size, but they are not replica of each other, their elements are different at different nodes. At some point of time, each node needs the matrix from all other nodes. Thus, each node shall send its own matrix at the same receive data from all other nodes.

I would be really grateful to you valuable suggestions and maybe solutions.

Improve this question
$\endgroup$
6
  • $\begingroup$ Using Isend and Recv is typically the way to do this. Why exactly do you think this may be insufficient? Trying to optimize something that may not need to be optimized is a common mistake, and the old adage "premature optimization is the root of all evil" applies. If using Isend/Recv is not sufficient, then you will find out about this during benchmarking -- but not by just thinking that it might. $\endgroup$
    – Wolfgang Bangerth
    Feb 8, 2016 at 6:18
  • 3
    $\begingroup$ You have not made it at all clear what communication pattern you are trying to implement. Code or equations would help. $\endgroup$
    – Jeff Hammond
    Feb 9, 2016 at 3:38
  • $\begingroup$ I think this is not an MPI problem, but an algorithm problem. Using isend and recv is the way to go. But When to call them, how, etc. could make a difference. But you need to explain better what are doing. What algorithm are you using? what problem? what equations? some code would help too. $\endgroup$
    – cauchi
    Feb 11, 2016 at 12:36
  • $\begingroup$ I would focus on bypassing the "each node needs the matrix from all other nodes". This is not usually the way that things work, especially in very big problems. If you really want to do that, then maybe you should try shared-memory parallelization. If you want to investigate your communication patterns deeper, you should use a tracing tool, like Vampir. $\endgroup$
    – MakisH
    Nov 9, 2016 at 8:43
  • $\begingroup$ With a lack of specifics on your particular problem, it's a bit difficult to offer suggestions. But on the off-chance that you're performing some kind of a distributed matrix-vector product A.x, you should really be distributing segments of x and operating on them with the local submatrices rather than communicating enormous arrays $\endgroup$
    – coolguy1000000
    Dec 4, 2017 at 14:57

2 Answers 2

Reset to default
1
$\begingroup$

MPI_ISEND and MPI_IRECV are commonly used for non-blocking communication between processes. If you want to sent data to all processes, collective operations viz. MPI_BCAST or MPI_IBCAST can be used.

Improve this answer
$\endgroup$
1
$\begingroup$

What you're describing (each node needs every other node's data) is exactly implemented by MPI_Allgather:

Diagram of the <code>MPI_Allgather</code> operation

You might expect MPI_Allgather to be more efficient than numerous MPI_ISEND/MPI_IRECV pairings since making MPI aware of your intended communication pattern could allow it to optimize information flow.

On the other hand, if you have wide variation in per-node calculation time, the MPI_ISEND/MPI_IRECV pairings would allow you to begin transferring information immediately as the nodes complete their operations, which might reduce latency.

Some of these trade-offs are discussed in the answers to this question.

The best thing to do is to profile your communication using, say, Vampir, and address the problems you find. Otherwise, try both and compare their performance.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.