I was trying to understand more in detail how a blocking MPI all to all really work. Of course, the actual all to all traffic is pretty straightforward, what I can't understand is how it is able to "keep" blocking until the collective is done.

I currently have two theories:

  1. For each rank the call is blocking until it has fully received the data from all its posted receives. But this seems counterintuitive to me as not all ranks will release at the same time then.
  2. Each rank keep track of all the other ranks and it only mark the collective as finished when all other ranks signals to it (I guess with some sort of broadcast) that they have received all the expected data.

Note that I am a noob to MPI and I could be completely wrong, so feel free to school me or point to me to a clear resource where it is explained.


1 Answer 1


Your theory 1 is correct. Why do you think all ranks "release" at the same time? They don't. Why would this even be necessary? Your mental model of MPI is incorrect. Processes run completely independently, except for when they synchronize. But a sending process can (often) continue, even if the receive is not finished. In an allreduce, any process that has the data can continue working. Time synchronization is basically never needed.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.