When is MPI_Wait necessary for non-blocking calls?

Question

I hope this is not too off-topic here -- I've asked it on SO but I'm hoping I'll get better answers here!

---

I'm a little bit confused about when I should call MPI_Wait (or other variants such as: MPI_Waitall, MPII_Waitsome, etc). Consider the following situations: (Note: pseudo code)

Case (1)

MPI_Isend (send_buffer, send_req);    
// Do local work
MPI_Probe (recv_msg);
MPI_Irecv (recv_buffer, recv_req);
// wait for msgs to finish
MPI_Wait (recv_req);   // <--- Is this needed?
MPI_Wait (send_req);   // <--- How about this?

So my confusion stems from MPI_Probe in this case. Since this is a blocking call, wouldn't that essentially mean it blocks the caller until message is received? If this is the case, then I think MPI_Waits are unnecessary here.

How about the following case?

Case (2)

MPI_Isend (send_buffer, send_req);    
// Do local work
MPI_Probe (recv_msg);
MPI_Recv (recv_buffer);
// wait for msgs to finish
MPI_Wait (send_req);   // <--- Is this necessary?

Similar to the first case but MPI_Irecv is replaced with its blocking version. In this case, the message is definitely received by the time MPI_Wait is called which means MPI_Isend must have been finished ...

Also as a separate question, what do we mean when we say MPI_Probe is blocking? Does it block until all of the message is received by the process or does it only block until "meta-data" (such as msg size, sender rank, etc) is received? In other words is MPI_Probe + MPI_Irecv any better than MPI_Probe + MPI_Recv ?

Answer 1

Bill answered the first part, so I'll only answer the second question. An MPI send is blocking if it does not return until it is safe to modify the send buffer and a receive is blocking if it does not return until the receive buffer contains the newly-received message. In practice, outside of buffered sends (thanks, Hristo Iliev), this implies that communication may be required before returning. For example, MPI_Send is blocking because it cannot complete before the message has been buffered or sent. Implementations will usually "eagerly" buffer short messages, in which case MPI_Send appears to return immediately. This means that code looks like:

MPI_Send(&x,count,MPI_INT,(rank+1)%size,1,comm);
MPI_Recv(&y,count,MPI_INT,(size+rank-1)%size,1,comm,MPI_STATUS_IGNORE);

is expected to deadlock for large messages, although it will likely succeed for small enough messages. Nonblocking sends and receives return MPI_Requests that must be completed before buffers are accessed/modified.

Some operations are a bit different from sends and receives. MPI_Probe is blocking because it does not return until a message has been found, although the message need not have been received yet. MPI_Iprobe is non-blocking in that it always returns even if there is no message.

Answer 2

MPI_Probe allows you to test for a message without actually receiving it. You must complete all non-blocking communications with an appropriate communications completion fuction like MPI_Wait and friends, otherwise the runtime will not free up internal resources associated with the communications leading to resource leaks and other problems. For example, you may not reuse the send message buffer for anything else until the communication has been completed with an appropriate completion call.

Answer 3

Every time that you use an MPI_Request in an MPI function, you either have to wait for it to finish using any of the MPI_Wait functions or keep on doing MPI_Test on it until the flag returns non-zero.

This includes all functions that start with MPI_I (non-blocking sends, receives and collectives) and functions that cancel a some non-blocking operation e.g.

int i = 0;
MPI_Request req = MPI_REQUEST_NULL;
MPI_Isend( &i, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &req );
MPI_Cancel( &req );
MPI_Wait( &req, MPI_STATUS_IGNORE ); // you have to wait

Answer 4

You are right that there are situations where you can determine by reasoning that an Isend or Irecv has been concluded, and so you don't have to do a Wait semantically. In that case, the only reason for doing the wait is that the "I" operation allocates an MPI_Request object and you need to deallocate it. So the wait prevents memory leaks. You can also use MPI_Request_free to deallocate the request object.