# MPI policy for multiple asynchronous transfers

What is the policy of multiple overlapping asynchronous transfers in MPI?

I have a program with several open asynchronous irecv operations. I find that transfers that could take place (the corresponding isend has been called) wait on other transfers that aren't yet ready (the corresponding isend hasn't yet been called). To be clear this inefficiency does not stem from network contention; my network is unnecessarily idle.

My program looks like the following:

### Machine 1

call irecv(variable A from machine 2)
call irecv(variable B from machine 2)
call irecv(variable C from machine 2)
call wait(variable C from machine 2)
call do_important_work_with(variable C)
....


### Machine 2

call isend(variable C to machine 1)
call isend(variable B to machine 1)
call do a bunch of costly work
call isend(variable A to machine 1)
....


### Problem

The transfer of C seems to be unnecessarily blocked by the transfer of A.

I find that the wait on variable C on Machine 1 does not complete until after the costly work on Machine 2 completes. This is unfortunate because this transfer could have started at the beginning of my program. It seems to wait unnecessarily for the transfer of A to complete.

### Questions

In particular I have a computation like the following.

• Is this expected?
• What is the policy of multiple overlapping asynchronous transfers?
• Can this be avoided without rearranging my code (is there some relevant internal setting)?
• How big are the transfers? Transfers with the same signature are required to happen in order. Do you use different tags for the different transfers? Also, it should not matter which MPI stack you use. The semantics of the ordering of transfers are defined by the MPI standards. – Bill Barth Jul 29 '13 at 23:38
• Transfers are large (around 1MB) and have the same size/source/destination (is this the signature?). They have different tags. – MRocklin Jul 29 '13 at 23:40
• Different tags should allow them to go in any order, but the hardware has to actually move the data, and it can't really do that in parallel. So, if it's a large message, you may be waiting on the underlying hardware to copy A and B into internal buffers or DMA it to the NIC (depending on what hardware you have). I would recommend altering the order that you post the receives in and also try using a different stack (MPICH, MVAPICH, Intel MPI, etc.) depending on your hardware. Also, you might try turning on progress threads. – Bill Barth Jul 30 '13 at 14:05
• If you have this sort of communication pattern, already over Ethernet, I would strongly recommend using zmq instead of mpi. – meawoppl Aug 5 '13 at 15:38

There's no guarantee in the standard that any progress is made on the non-blocking sends until you actually call MPI_WAIT. It's a perfectly valid implementation to just queue up the operations and when you call MPI_WAIT, all of the MPI_ISEND operations complete at once. In reality, they usually tend to get a chance to progress anytime you enter the MPI library and if you enable asynchronous progress threads, they have a better chance of progressing in the background.