What is the policy of multiple overlapping asynchronous transfers in MPI?
I have a program with several open asynchronous irecv
operations. I find that transfers that could take place (the corresponding isend
has been called) wait on other transfers that aren't yet ready (the corresponding isend
hasn't yet been called). To be clear this inefficiency does not stem from network contention; my network is unnecessarily idle.
My program looks like the following:
Machine 1
call irecv(variable A from machine 2)
call irecv(variable B from machine 2)
call irecv(variable C from machine 2)
call wait(variable C from machine 2)
call do_important_work_with(variable C)
....
Machine 2
call isend(variable C to machine 1)
call isend(variable B to machine 1)
call do a bunch of costly work
call isend(variable A to machine 1)
....
Problem
The transfer of C
seems to be unnecessarily blocked by the transfer of A
.
I find that the wait
on variable C
on Machine 1 does not complete until after the costly work on Machine 2 completes. This is unfortunate because this transfer could have started at the beginning of my program. It seems to wait unnecessarily for the transfer of A
to complete.
Questions
In particular I have a computation like the following.
- Is this expected?
- What is the policy of multiple overlapping asynchronous transfers?
- Can this be avoided without rearranging my code (is there some relevant internal setting)?
- Where should I go to learn more about MPI's policy for multiple live transfers?