8
$\begingroup$

What is the policy of multiple overlapping asynchronous transfers in MPI?

I have a program with several open asynchronous irecv operations. I find that transfers that could take place (the corresponding isend has been called) wait on other transfers that aren't yet ready (the corresponding isend hasn't yet been called). To be clear this inefficiency does not stem from network contention; my network is unnecessarily idle.

My program looks like the following:

Machine 1

call irecv(variable A from machine 2)
call irecv(variable B from machine 2)
call irecv(variable C from machine 2)
call wait(variable C from machine 2)
call do_important_work_with(variable C)
....

Machine 2

call isend(variable C to machine 1)
call isend(variable B to machine 1)
call do a bunch of costly work
call isend(variable A to machine 1)
....

Problem

The transfer of C seems to be unnecessarily blocked by the transfer of A.

I find that the wait on variable C on Machine 1 does not complete until after the costly work on Machine 2 completes. This is unfortunate because this transfer could have started at the beginning of my program. It seems to wait unnecessarily for the transfer of A to complete.

Questions

In particular I have a computation like the following.

  • Is this expected?
  • What is the policy of multiple overlapping asynchronous transfers?
  • Can this be avoided without rearranging my code (is there some relevant internal setting)?
  • Where should I go to learn more about MPI's policy for multiple live transfers?
Improve this question
$\endgroup$
4
  • $\begingroup$ How big are the transfers? Transfers with the same signature are required to happen in order. Do you use different tags for the different transfers? Also, it should not matter which MPI stack you use. The semantics of the ordering of transfers are defined by the MPI standards. $\endgroup$
    – Bill Barth
    Jul 29 '13 at 23:38
  • $\begingroup$ Transfers are large (around 1MB) and have the same size/source/destination (is this the signature?). They have different tags. $\endgroup$
    – MRocklin
    Jul 29 '13 at 23:40
  • $\begingroup$ Different tags should allow them to go in any order, but the hardware has to actually move the data, and it can't really do that in parallel. So, if it's a large message, you may be waiting on the underlying hardware to copy A and B into internal buffers or DMA it to the NIC (depending on what hardware you have). I would recommend altering the order that you post the receives in and also try using a different stack (MPICH, MVAPICH, Intel MPI, etc.) depending on your hardware. Also, you might try turning on progress threads. $\endgroup$
    – Bill Barth
    Jul 30 '13 at 14:05
  • $\begingroup$ If you have this sort of communication pattern, already over Ethernet, I would strongly recommend using zmq instead of mpi. $\endgroup$
    – meawoppl
    Aug 5 '13 at 15:38
6
$\begingroup$

There's no guarantee in the standard that any progress is made on the non-blocking sends until you actually call MPI_WAIT. It's a perfectly valid implementation to just queue up the operations and when you call MPI_WAIT, all of the MPI_ISEND operations complete at once. In reality, they usually tend to get a chance to progress anytime you enter the MPI library and if you enable asynchronous progress threads, they have a better chance of progressing in the background.

As for the signature issue, MPI guarantees that messages on the same communicator to the same ranks will be received in the same order in which they were sent.

From the MPI Standard version 3.0:

Order Messages are non-overtaking: If a sender sends two messages in succession to the same destination, and both match the same receive, then this operation cannot receive the second message if the first one is still pending. If a receiver posts two receives in succession, and both match the same message, then the second receive operation cannot be satisfied by this message, if the first one is still pending.

This doesn't say anything about how the implementation chooses to send the messages, but at least they will be received in the correct order.

My advice would be to first make sure you have progress threads enabled, then make sure you're calling wait where you really need the messages sent (though with progress threads, you'll most likely be fine).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.