4
$\begingroup$

A typical way of dealing with I/O in MPI parallel programs is to either read all data to a single node and dispatch to the other nodes accordingly, or send all data to a single node and write from this node.

I currently use blocking communication.

Since the "master" process can only communicate with one node at a time, most nodes are stuck in the blocking communication step : they essentially "do nothing" while they wait for the master to send/receive. This is completely understandable.

However, when I have a look at CPU usage using "top", all copies of the program have a full 99 - 100% CPU usage. This means that (contrary to my initial guess) waiting for the other side of the communication is not an "idle task" taking up a fairly low amount of CPU, but rather a "busy" one.

How can I reduce the CPU usage for these "waiting" nodes ? Is non-blocking communication relevant here ?

Thanks for any answer,

Improve this question
$\endgroup$
3
  • $\begingroup$ Just a couple of clarification/confirmation points: 1) Are you sure that those other nodes have already reached that place in the code where blocking communication is supposed to happen? (simplest, you have print statements around the communication block) 2) Are you using MPI_Send/MPI_Recv or the communication is slightly more advanced? 3) there might be an advantage of using MPI_Gather or other communication routines depending on your exact goal, so if you describe the exact communication pattern and the purpose in more detail, it might help. $\endgroup$
    – Anton Menshov ♦
    Commented Nov 27, 2019 at 15:54
  • 1
    $\begingroup$ 1) Yes, definately 2) Yes, simple send/recv 3) The communication pattern for the input goes something like : Rank 0 : Generate own data - print own data - Receive data from proc 1 - print data from proc 1 --- Receive data from proc n - print data from proc n. Rank != 0 : Generate data - send data. $\endgroup$
    – G. Fougeron
    Commented Nov 28, 2019 at 9:52
  • $\begingroup$ I am trying to replicate this slightly unexpected behaviour. Are you using mpich, openmpi or something else? $\endgroup$
    – Anton Menshov ♦
    Commented Nov 28, 2019 at 16:41

4 Answers 4

Reset to default
3
$\begingroup$

I can confirm that with blocking waits the receiving processes are spinning at 100 percent. They are probably in a polling loop of sorts.

  1. With non-blocking receives you get exactly the same behavior: then the polling loop is in the MPI_Wait call.
  2. Why does this bother you? Do you worry about the cost of electricity? If your program is waiting there, then clearly it has nothing else to do.
  3. If your program does have something to do in the meantime you can of course let it do that, and occasionally do MPI_Test on the outstanding requests.
Improve this answer
$\endgroup$
6
$\begingroup$

It depends on the communication settings you use for MPI. In blocking communication, MPI has three wait modes.

  1. Aggressive busy wait. This is a kind of default mode. Open MPI, at least, uses this when it thinks it is exactly- or under-subscribed (number of processes<=number of processesors). In this mode processes will never voluntarily give up the processor to other processes. OpenMPI spins in tight loops attempting to make message passing progress as fast as possible. Other processes to not get any CPU cycles and never progress. Force this mode with: mpirun -np N --mca mpi_yield_when_idle 0 ./a.out

  2. Degraded busy wait. Useful if you are oversubscribed (number of processes>number of processors), processes frequently yield the processor, thereby allowing multiple processes to progress. Slightly slower than aggressive mode if you are not oversubscribed. Force this mode with: mpirun -np N --mca mpi_yield_when_idle 1 ./a.out

  3. Polling. This one you have to set up yourself by, e.g., calling MPI_Iprobe() in a loop with a sleep call.

Improve this answer
$\endgroup$
2
$\begingroup$

I am pretty sure I've heard before that MPI wait operations (in most MPI packages) are implemented using a spin lock -- i.e., the processor is stuck in a loop (running at full speed) checking over and over again whether any information has come in.

I'm not an MPI designer and so don't have the background to say why that decision was made -- but heard that it was for good reasons.

Improve this answer
$\endgroup$
0
$\begingroup$

spinlocks are bad because other threads in other processes could have used the CPUs. And yes, from the OS point of view, the polling thread is always in the running state which prevents the CPU from tuning down the clock to save electricity. An ideal implementation would make the thread waiting on some sync object and be notified when data arrives.

Improve this answer
$\endgroup$
2
  • $\begingroup$ This is mpi, therefore processes, not threads $\endgroup$
    – Ian Bush
    Commented Sep 24 at 8:27
  • $\begingroup$ Odeal in some spplications, but slower in general. In supercomputing, you often want to run as fast as possible, although energy consumption is now an inportant consideration also. $\endgroup$
    – Vladimir F Героям слава
    Commented Sep 24 at 8:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.