A reduction implemented using
MPI_Allreduce() is reproducible as long as you use the same number of processors, provided the implementation observed the following note appearing in Section 5.9.1 of the MPI-2.2 standard.
Advice to implementors. It is strongly recommended that
MPI_REDUCE be implemented so that the same result be obtained whenever the function is applied on the
same arguments, appearing in the same order. Note that this may prevent optimizations that take advantage of the physical location of processors. (End of advice to
If you need to guarantee reproducibility at all costs, you can follow the guidelines in the next paragraph:
Advice to users. Some applications may not be able to ignore the non-associative nature of floating-point operations or may use user-defined operations (see Section 5.9.5)
that require a special reduction order and cannot be treated as associative. Such
applications should enforce the order of evaluation explicitly. For example, in the
case of operations that require a strict left-to-right (or right-to-left) evaluation order, this could be done by gathering all operands at a single process (e.g., with
MPI_GATHER), applying the reduction operation in the desired order (e.g., with
MPI_REDUCE_LOCAL), and if needed, broadcast or scatter the result to the other
processes (e.g., with
MPI_BCAST). (End of advice to users.)
In the broader scheme of things, efficient algorithms for most applications take advantage of locality. Since the algorithm is really different when run on a different number of processes, it is just not practical to exactly reproduce results when run on a different number of processes. A possible exception is multigrid with damped Jacobi or polynomial (e.g. Chebyshev) smoothers, where it is possible for this simple method to perform very well.
With the same number of processes, it is often beneficial to performance to process messages in the order that they are received (e.g. using
MPI_Waitany()), which introduces non-determinism. In such cases, you can implement two variants, the fast one that receives in any order and a "debug" one that receives in a static order. This requires that all underlying libraries are also written to offer this behavior.
For debugging in some cases, you can isolate part of a computation that does not offer this reproducible behavior and perform that redundantly. Depending on how the components were designed, that change may be a tiny amount of code or very intrusive.