# Causes for different results for different number of nodes in MPI

I am trying to develop what might be called legacy Fortran code from my group. Upon testing, I found that the program (which takes about a day to run on 20 processes) works when run on a single node (20 processes) or in serial (though it takes a long time), but if I increase to several nodes the results are notably, non-negligibly different (though it does not crash). Looking around for what might cause this, it seems like the most common is some overwriting of shared memory (e.g. opening a file without specifying only rank 0). Is this right? What else might cause what seems to be inter-node communication problems? For what it's worth, it is just this library; we run plenty of other MPI programs on the same cluster with the same compilers without this issue. I find it very odd, since as far as I've seen you never explicitly handle inter-node communication in MPI, just between different processes.

My guess is there is a bug in the code; what kind of functions/common pitfalls with MPI would lead to this kind of behaviour? I haven't seen this kind of issue before (and was unaware it was even possible) and would like to know how to debug it, preferably without hogging several nodes from a shared cluster. I also want to write unit tests that can prevent this sort of thing from happening again, but I have no idea how to write tests for things that only appear on several nodes. It would also be especially helpful if someone could share/link to a minimal program that exhibits this behaviour.

It's a large complicated codebase that I don't fully understand, but it is written modularly. I am interested in trying to write unit tests for it to pinpoint the issue (not to mention improve maintainability). How can I test for an error that only shows up with several nodes? I know pFUnit has MPI support and it looks like they have a mock MPI which you can run quickly to "pretend" you're in parallel without bothering your cluster. However, it looks like I can only mock the number of processes and not the number of nodes. Can I do this with pFUnit? Otherwise, is there some other unit testing framework that would do the trick? Currently there are a few tests written with FRUIT but this looks like a rather minimal framework for testing MPI programs.

Edit: I ran some more tests to make sure it wasn't just the larger number of processes. Indeed, it is the number of nodes changing the results. I ran the script for 20 processes distributed among 1,2,4 nodes and the results were all slightly different. I reran them a few times and they are consistent.

Edit 2: The code uses shared memory. While I do not understand how, there are comments labelling variables like ! MPI shared memory window. There is also a small snippet of code that uses openMP !\$omp parallel do with a few shared variables. However, commenting out the openMP directives gives the same behaviour.

• Does the code reproduce manufactured solutions?
– Bort
Jul 8 at 7:49
• It is not possible to figure out your problem without looking at your code. Try to simplify your code and post here.
– Misa
Jul 9 at 22:44

There is no way for anyone to tell you where the issue comes from without access to the actual code. But here are common reasons why the output (legitimately) differs for different numbers of processes:

• If a code has to enumerate objects to be worked on (for example in "embarrassingly parallel" problems, but also degrees of freedom in mesh-based computations), then a common strategy is to let each process enumerate the objects it has stored locally, starting at the last index used by the previous process. That means that the order of objects depends on the number of processes, and many (iterative) algorithms will produce results that depend on the order if they stop whenever a certain tolerance is reached, rather than driving the iteration error to zero.

• Floating point operations are not associative, i.e., in general $$a+b+c \neq a+c+b$$. That means that reduction operations typically produce slightly different results based on how many processes you have, and the end result of computations might therefore also different depending on the number of processes.

Of course, there is also the possibility that there simply is a bug in the code.

To find where the problem lies, you need to find the first place in the flow of computations where you get different results. In other words, don't focus on the end result, but check intermediate points of the program -- say, after the first iteration or time step.

• Thanks for the reply, of course I wasn't expecting some exact answer. It sounds like you don't address the multi-node problem, just the many processors. The scenarios you give sound like they would still be present in the multi-core parallel calculation on one node. I am trying to find where in the code it happens via unit testing, but again it is seemingly only for many nodes, so it would be nice to be able to mock that.
– tmph
Jul 7 at 21:37
• I haven't seen a case of "legitimately different" results where it matters whether the MPI jobs run on one or on multiple nodes. If you get different results based on whether you run 20 jobs on one node or whether you run 20 jobs on two nodes, then there is a bug. Jul 8 at 2:17
• The only difference between using one and multiple nodes is the ability/inability of using shared memory regions. Few codes do that. Jul 8 at 2:17
• I ran a few more tests. 20 cores distributed among 1, 2, or 4 nodes all gave notably different results. I suppose then somewhere it is a shared memory issue. Is there any standard way to pinpoint such an issue/bug?
– tmph
Jul 8 at 15:02
• Does your code even use shared memory? Jul 8 at 16:14