Causes for different results for different number of nodes in MPI

Question

I am trying to develop what might be called legacy Fortran code from my group. Upon testing, I found that the program (which takes about a day to run on 20 processes) works when run on a single node (20 processes) or in serial (though it takes a long time), but if I increase to several nodes the results are notably, non-negligibly different (though it does not crash). Looking around for what might cause this, it seems like the most common is some overwriting of shared memory (e.g. opening a file without specifying only rank 0). Is this right? What else might cause what seems to be inter-node communication problems? For what it's worth, it is just this library; we run plenty of other MPI programs on the same cluster with the same compilers without this issue. I find it very odd, since as far as I've seen you never explicitly handle inter-node communication in MPI, just between different processes.

My guess is there is a bug in the code; what kind of functions/common pitfalls with MPI would lead to this kind of behaviour? I haven't seen this kind of issue before (and was unaware it was even possible) and would like to know how to debug it, preferably without hogging several nodes from a shared cluster. I also want to write unit tests that can prevent this sort of thing from happening again, but I have no idea how to write tests for things that only appear on several nodes. It would also be especially helpful if someone could share/link to a minimal program that exhibits this behaviour.

It's a large complicated codebase that I don't fully understand, but it is written modularly. I am interested in trying to write unit tests for it to pinpoint the issue (not to mention improve maintainability). How can I test for an error that only shows up with several nodes? I know pFUnit has MPI support and it looks like they have a mock MPI which you can run quickly to "pretend" you're in parallel without bothering your cluster. However, it looks like I can only mock the number of processes and not the number of nodes. Can I do this with pFUnit? Otherwise, is there some other unit testing framework that would do the trick? Currently there are a few tests written with FRUIT but this looks like a rather minimal framework for testing MPI programs.

Edit: I ran some more tests to make sure it wasn't just the larger number of processes. Indeed, it is the number of nodes changing the results. I ran the script for 20 processes distributed among 1,2,4 nodes and the results were all slightly different. I reran them a few times and they are consistent.

Edit 2: The code uses shared memory. While I do not understand how, there are comments labelling variables like ! MPI shared memory window. There is also a small snippet of code that uses openMP !$omp parallel do with a few shared variables. However, commenting out the openMP directives gives the same behaviour.

Answer 1

There is no way for anyone to tell you where the issue comes from without access to the actual code. But here are common reasons why the output (legitimately) differs for different numbers of processes:

• If a code has to enumerate objects to be worked on (for example in "embarrassingly parallel" problems, but also degrees of freedom in mesh-based computations), then a common strategy is to let each process enumerate the objects it has stored locally, starting at the last index used by the previous process. That means that the order of objects depends on the number of processes, and many (iterative) algorithms will produce results that depend on the order if they stop whenever a certain tolerance is reached, rather than driving the iteration error to zero.

• Floating point operations are not associative, i.e., in general $a+b+c \neq a+c+b$. That means that reduction operations typically produce slightly different results based on how many processes you have, and the end result of computations might therefore also different depending on the number of processes.

Of course, there is also the possibility that there simply is a bug in the code.

To find where the problem lies, you need to find the first place in the flow of computations where you get different results. In other words, don't focus on the end result, but check intermediate points of the program -- say, after the first iteration or time step.