First, the exact answer depends on: (1) usage, i.e. function input arguments, (2) MPI implementation quality and details, and (3) the hardware you're using. Often, (2) and (3) are related, such as when the hardware vendor optimizes MPI for their network.
In general, fusing MPI collectives is better for smaller messages, since start-up costs can be nontrivial and the synchronization entailed by blocking collectives should be minimized if there is variation in compute time between calls. For larger messages, the goal should be to minimize the amount of data being sent.
For example, in theory, MPI_Reduce_scatter_block
should be better than MPI_Reduce
followed by MPI_Scatter
, although the former is often implemented in terms of the latter, such that there is no real advantage. There's a correlation between implementation quality and frequency of usage in most implementations of MPI, and vendors obviously optimize those functions for which this is required by the machine contract.
On the other hand, if one is on a Blue Gene, doing MPI_Reduce_scatter_block
using MPI_Allreduce
, which does more communication than MPI_Reduce
and MPI_Scatter
combined, is actually quite a bit faster. This is something I recently discovered and is an interesting violation of the principle of performance self-consistency in MPI (this principle is described in more detail in "Self-Consistent MPI Performance Guidelines").
In the specific case of scatter+gather versus allgather, consider that in the former, all the data must go to and from a single process, which makes it the bottleneck, whereas in the allgather, data can flow in and out of all ranks immediately, because all ranks have some data to send to all other ranks. However, sending data from all nodes at once is not necessarily a good idea on some networks.
Finally, the best way to answer this question is to do the following in your code and answer the question by experiment.
#ifdef TWO_MPI_CALLS_ARE_BETTER_THAN_ONE
MPI_Scatter(..)
MPI_Gather(..)
#else
MPI_Allgather(..)
#endif
An even better option is to have your code measure it experimentally during the first two iterations, then use whichever is faster for the remaining iterations:
const int use_allgather = 1;
const int use_scatter_then_gather = 2;
int algorithm = 0;
double t0 = 0.0, t1 = 0.0, dt1 = 0.0, dt2 = 0.0;
while (..)
{
if ( (iteration==0 && algorithm==0) || algorithm==use_scatter_then_gather )
{
t0 = MPI_Wtime();
MPI_Scatter(..);
MPI_Gather(..);
t1 = MPI_Wtime();
dt1 = t1-t0;
}
else if ( (iteration==1 && algorithm==0) || algorithm==use_allgather)
{
t0 = MPI_Wtime();
MPI_Allgather(..);
t1 = MPI_Wtime();
dt2 = t1-t0;
}
if (iteration==1)
{
dt2<dt1 ? algorithm=use_allgather : algorithm=use_scatter_then_gather;
}
}
MPI_Scatter
followed byMPI_Gather
does not provide the same communication semantic asMPI_Allgather
. Perhaps there is redundancy involved when you express the operation in either way? $\endgroup$ – Jed Brown Mar 27 '12 at 14:57MPI_Gather
followed by aMPI_Bcast
? $\endgroup$ – Aron Ahmadia Mar 27 '12 at 15:03