# When is MPI_Wait necessary for non-blocking calls?

I hope this is not too off-topic here -- I've asked it on SO but I'm hoping I'll get better answers here!

I'm a little bit confused about when I should call MPI_Wait (or other variants such as: MPI_Waitall, MPII_Waitsome, etc). Consider the following situations: (Note: pseudo code)

Case (1)

MPI_Isend (send_buffer, send_req);
// Do local work
MPI_Probe (recv_msg);
MPI_Irecv (recv_buffer, recv_req);
// wait for msgs to finish
MPI_Wait (recv_req);   // <--- Is this needed?


So my confusion stems from MPI_Probe in this case. Since this is a blocking call, wouldn't that essentially mean it blocks the caller until message is received? If this is the case, then I think MPI_Waits are unnecessary here.

Case (2)

MPI_Isend (send_buffer, send_req);
// Do local work
MPI_Probe (recv_msg);
MPI_Recv (recv_buffer);
// wait for msgs to finish
MPI_Wait (send_req);   // <--- Is this necessary?


Similar to the first case but MPI_Irecv is replaced with its blocking version. In this case, the message is definitely received by the time MPI_Wait is called which means MPI_Isend must have been finished ...

Also as a separate question, what do we mean when we say MPI_Probe is blocking? Does it block until all of the message is received by the process or does it only block until "meta-data" (such as msg size, sender rank, etc) is received? In other words is MPI_Probe + MPI_Irecv any better than MPI_Probe + MPI_Recv ?

• You must always wait on or test nonblocking operations if you'd like your programs to be standard compliant and hence portable. The standard allows the implementation to postpone the actual data transmission until the wait/test call. Some MPI operations (other than wait/test) progress non-blocking operations but one should not rely on this behaviour. – Hristo Iliev Aug 24 '13 at 13:01

Bill answered the first part, so I'll only answer the second question. An MPI send is blocking if it does not return until it is safe to modify the send buffer and a receive is blocking if it does not return until the receive buffer contains the newly-received message. In practice, outside of buffered sends (thanks, Hristo Iliev), this implies that communication may be required before returning. For example, MPI_Send is blocking because it cannot complete before the message has been buffered or sent. Implementations will usually "eagerly" buffer short messages, in which case MPI_Send appears to return immediately. This means that code looks like:

MPI_Send(&x,count,MPI_INT,(rank+1)%size,1,comm);
MPI_Recv(&y,count,MPI_INT,(size+rank-1)%size,1,comm,MPI_STATUS_IGNORE);


is expected to deadlock for large messages, although it will likely succeed for small enough messages. Nonblocking sends and receives return MPI_Requests that must be completed before buffers are accessed/modified.

Some operations are a bit different from sends and receives. MPI_Probe is blocking because it does not return until a message has been found, although the message need not have been received yet. MPI_Iprobe is non-blocking in that it always returns even if there is no message.

• This is not true. Blocking means that return from the procedure indicates that the user is allowed to reuse the resources specified in the call, e.g. the data buffer (see §2.4 of the MPI standard). For example MPI_Bsend always buffers the message and is still a blocking call. – Hristo Iliev Aug 24 '13 at 12:55
• @HristoIliev You're completely right, I wasn't thinking about buffered-mode. I've updated and expanded the answer. – Jed Brown Aug 24 '13 at 15:59

MPI_Probe allows you to test for a message without actually receiving it. You must complete all non-blocking communications with an appropriate communications completion fuction like MPI_Wait and friends, otherwise the runtime will not free up internal resources associated with the communications leading to resource leaks and other problems. For example, you may not reuse the send message buffer for anything else until the communication has been completed with an appropriate completion call.

• I see. But still I have my other question. Does MPI_Probe block the process until ALL of the message is received or does it return as soon as it figures out the length of message and sending process? – mmirzadeh Aug 23 '13 at 21:03
• @GradGuy It only needs to determine the size. Some implementations might eagerly receive small messages, but quality implementations won't buffer large messages because that could use an excessive amount of memory. – Jed Brown Aug 23 '13 at 21:30
• @JedBrown That's what I was hoping for but could not find much info on the web. Thanks Jed. – mmirzadeh Aug 23 '13 at 21:43

Every time that you use an MPI_Request in an MPI function, you either have to wait for it to finish using any of the MPI_Wait functions or keep on doing MPI_Test on it until the flag returns non-zero.

This includes all functions that start with MPI_I (non-blocking sends, receives and collectives) and functions that cancel a some non-blocking operation e.g.

int i = 0;
MPI_Request req = MPI_REQUEST_NULL;
MPI_Isend( &i, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &req );
MPI_Cancel( &req );
MPI_Wait( &req, MPI_STATUS_IGNORE ); // you have to wait


You are right that there are situations where you can determine by reasoning that an Isend or Irecv has been concluded, and so you don't have to do a Wait semantically. In that case, the only reason for doing the wait is that the "I" operation allocates an MPI_Request object and you need to deallocate it. So the wait prevents memory leaks. You can also use MPI_Request_free to deallocate the request object.