1
$\begingroup$

Testing on an MPI scientific code for compressible flow dynamics I noticed that the results may depend on the number of processors used for the calculation. In fact, comparing the binary files they differ around 10.0E-10 ~ 10.0E-11

Since I am optimising the application and I am almost completely rewriting it from zero, I wonder if it is possible to make the results independent of the number of MPI processes.

  • If yes, how to proceed?

  • What are the things to avoid and what are the things to do?

|cite|improve this question
$\endgroup$
2
$\begingroup$

That is a very common problem with parallel computations. Technically, if the communication pattern within your MPI-parallelized code is complicated enough, even runs with the same number of processors might end up having slightly different results.

Now, the strategy to deal with that will differ a lot depending on what and how is being parallelized. One of the common sources of "results fluctuation", is the reduction (due to non-associativity of floating point operations).

There are several things you can do about that. First, look at this CompSci question on MPI reduction reproducibility. Jed's answer gives a nice overview of the problem regarding reduction and an advice to have stable results for the same and the different number of processors.

In short, do your calculations in parallel, until you need to "merge" them together (reduction). To merge them, gather them on one processor (commonly referred to as the master node) and perform the required set of operations. When doing the last step locally, you can be in charge of the operation order.

However, the reduction is not the only possible source: as your problem partitioning (mesh partitioning, etc.) might already introduce another source of "fluctuations". So, I would suggest

  • to reevaluate your criteria on independence on the number of MPI processes
  • if you decide you need it, to have both (more) stable, debug, but slower version, and standard one.
  • have some tolerance to which you would allow the results to be different.
|cite|improve this answer
$\endgroup$
  • $\begingroup$ I was aware that the Reduce / allReduce operations were not safe in terms of binary equality. This is why I eliminated all of them from the core of the calculation (also in order to increase the computational efficiency). Even without these operations MPI does not respect the equality of the tracks? In particular, I noticed that the MPI compiler (gnu mpif90) is much more susceptible to overflow / underflow than the serial compiler, is it by chance what is known? $\endgroup$ – John Snow Oct 21 '18 at 12:57
  • 1
    $\begingroup$ How do you eliminate reduction operations? Especially to increase efficiency? $\endgroup$ – Victor Eijkhout Oct 22 '18 at 22:37
0
$\begingroup$

The associativity issue is address in other answers (and see my comment); it mostly plays in collectives. Another thing to watch out for it not to use wildcard receives. If the order of incoming messages is indeterminate the associativity issue comes up again.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.