# What is an efficient way of notifying MPI processes of receiving messages?

In MPI, is there any built-in mechanism to notify a group of processes that they need to receive messages from other processes?

In my application every process needs to send data to a group of processes with known rank IDs (which potentially changes at each iteration), but they do not know the rank of the processes that will be receiving messages from. Is there a portable, preferably built-in, way of achieving this without querying each and every process?

You could try having all processors use MPI_IProbe or MPI_Probe with MPI_ANY_SOURCE to check if there are any receivable messages with a given tag. If there are matching messages, you can extract the senders rank from the returned status and call MPI_Recv immediately.

• Is calling MPI_Probe essentially cheaper than MPI_Recv ? Jul 5 '13 at 22:26
• @GradGuy With MPI_IProbe and MPI_Probe you don't actually receive any messages (you are just querying MPI to see if there are any receivable messages), so in that sense I suppose it is cheaper than MPI_Recv. Jul 7 '13 at 3:54
• @MatthewEmmett How receiver knows how many MPI_IProbe calls to post or how long keep probing? May 21 '17 at 14:54

Others have already proposed the various MPI_Probe variants but I'd like to point out one thing: MPI is not a remote procedure call, i.e., there are no ways to notify a process that some message has come in (e.g., by raising a signal). Messages are sent but if the receiving process doesn't actually go look for them, then nothing will happen. As such, the answer to your question is "no", but a process can actively figure out whether a message has been sent to it.

• In theory, MPI_Probe can be descheduled by the kernel such that it is passive until used, which is not too different from how IBM implements real RPCs on Blue Gene (where the OS and HW collaborate to implement very efficient network-initiated interrupts that can call arbitrary user-defined callbacks). Of course, no implementation I know about actually does this with blocking MPI calls... Jul 6 '13 at 20:06
• The point I wanted to make is not so much about how it's actually implemented, but about the mental model one should have when thinking about MPI. MPI is purely a passive concept: you can't access data from another process, you can't expect a signal to be raised if an incoming message arrives; everything only ever happens if you actively send a message or look whether one came in. Jul 7 '13 at 18:46
• You're right about a signal but RMA gives you access to remote data and implementations are supposed to provide passive progress. Jul 10 '13 at 19:58
• Yes, you are talking about what MPI 3 calls "one-sided communication". That's a fairly recent addition, and likely a rarely used one. But I think it doesn't take away from the mental model I laid out above. Jul 11 '13 at 11:39
• MPI 2.0 had RMA ten years ago and one can implement e.g. put_with_notify using those features. I agree that MPI does not have remote procedure calls aka active messages (yet) but no non-IBM machine implements those without polling, so we're not really talking about MPI any more if we're criticizing the need for polling. Jul 14 '13 at 19:13

http://mpitutorial.com/tutorials/dynamic-receiving-with-mpi-probe-and-mpi-status/ has a tutorial describing the use of MPI_Probe that might be useful to you.

If the senders know to whom they are sending but the receivers don't know from whom they are receiving, just post your receives from MPI_ANY_SOURCE. That will match the receive with any process that sends a message to it.

If the receivers might not actually receive anything at all, they can post an iRecv. Whenever a process is done sending its messages, it can post am Ibarrier. When the Ibarrier completes because all processes have entered it (and called MPI_WAIT on the request object), everyone knows that the communication is done and they can cancel their previous Irecv.

• In what order does that receive the message? I need to send back the senders result of some calculation ... Jul 3 '13 at 23:57
• The receives will be in order per sending process. If rank 1 sends messages A - B - C in that order, they will arrive in that order. There's no guarantee of the ordering between the processes. Jul 3 '13 at 23:58
• I see ... but is it guaranteed that the messages from processes are not mixed together? Jul 3 '13 at 23:59
• No. The messages between processes can arrive in any order. Jul 4 '13 at 0:38
• MPI_Cancel is a horrible function and should be used as infrequently as possible. I highly endorse variants of Probe instead. Jul 4 '13 at 1:48