8
$\begingroup$

I have a bunch of MPI processes exchanging request messages back and forth. Processes do not know which other processes will send them messages, or how many. Given this situation, I want an efficient way to know whether all other processes consider themselves done sending messages.

This would be accomplished perfectly by the following nonblocking version of MPI_Barrier, which we'll call MPI_Ibarrier:

int MPI_Ibarrier(MPI_Comm comm, MPI_Request* request);

MPI_Ibarrier would return immediately, and standard operations on the request object would let us know when the barrier has been reached by everyone.

Is there a way to simulate this behavior efficiently in MPI 2 (i.e., without official nonblocking collectives)?

Improve this question
$\endgroup$
3
  • $\begingroup$ And, indeed, MPI_Ibarrier is all over Google in reference to the MPI 3 working group. Now we just need the future... $\endgroup$ – Geoffrey Irving Jul 20 '12 at 4:13
  • $\begingroup$ I don't see how this would work. Presumably you're planning on calling MPI_Ibarrier on a process when it's done with messages for this iteration and then calling the MPI_Wait when? If tasks are going to go on after MPI_Ibarrier, how far will they be allowed to get before they have to call MPI_Wait? If they're allowed to get however far that is before they have to know that all procs are done, why not just put MPI_Barrier at that point? IME, MPI_Barrier is almost always wrong to put in your code except for debugging purposes. There's almost always a more efficient way of structuring the code. $\endgroup$ – Bill Barth Jul 20 '12 at 12:05
  • $\begingroup$ In this case I believe MPI_Ibarrier is the optimal way, since any messages from the next communication epoch cannot be started without deallocating a very large chunk of memory and reallocating a new one. The barrier is needed to know when this memory can be deallocated, not directly when future messages can be sent. $\endgroup$ – Geoffrey Irving Jul 20 '12 at 14:29
12
$\begingroup$

Remarkably, MPI_Ibarrier is a very useful routine. For example, you can deliver an unstructured round of messages to ranks that do not know how many messages to receive by sending with MPI_Issend (yes, a rare use of synchronous send), then entering a loop of alternating MPI_Testall (to see if the sends completed) and MPI_Iprobe (to process incoming messages). When the sends complete, you post MPI_Ibarrier and alternative testing the barrier an probing for incoming messages. Torsten Hoefler has a paper on this where he proves communication optimality, see Algorithm 2: http://unixer.de/publications/img/hoefler-dsde-protocols.pdf

Note that a barrier does not ensure that point-to-point messages or other nonblocking collectives posted before the barrier have completed. If you want them to complete, you have to ensure that they have completed before posting the barrier. As Bill says, (blocking) MPI_Barrier is incorrect/unnecessary in most cases. An exception is communicating through side channels such as the file system.

Although there is not a similarly performant way to simulate MPI_Ibarrier with MPI-2, MPICH2 provides MPIX_Ibarrier in the 1.5 branch (current). Vendor networks generally support these operations so the vendor implementations (which are generally derived from MPICH2) just need an interface. As soon as their patches move to 1.5, MPI_Ibarrier and other nonblocking collectives should be supported. The Open MPI development branch has an implementation of MPI_Ibarrier based on libNBC.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I think you can implement that with a flag message/special tag that's more efficient than a barrier of any sort could possibly be. Do you have a link to Hoefler's paper? $\endgroup$ – Bill Barth Jul 20 '12 at 12:49
  • $\begingroup$ @BillBarth Remember that this is unstructured and you don't know how many messages you need to receive. Who would send the tag and to whom? You'll end up implementing your own Ibarrier over point-to-point, which will be slower than a native implementation. The paper is probably on his website. We talked about it a few months ago. $\endgroup$ – Jed Brown Jul 20 '12 at 12:54
  • $\begingroup$ Implementing my own tree reduction over point-to-point is exactly what I ended up doing. There are only 36 barrier calls in the entire program, so the slowdown shouldn't be problematic. $\endgroup$ – Geoffrey Irving Jul 20 '12 at 14:34
8
$\begingroup$

See http://www.unixer.de/research/nbcoll/, especially http://www.unixer.de/research/nbcoll/libnbc/, which is the reference implementation that sits on top of MPI-1. As Jed said, MPICH2 1.5 already has NBC and the implementation is similar to Torsten's (although almost certainly faster due to lower overhead).

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.