I am trying to develop what might be called legacy Fortran code from my group. Upon testing, I found that the program (which takes about a day to run on 20 processes) works when run on a single node (20 processes) or in serial (though it takes a long time), but if I increase to several nodes the results are notably, non-negligibly different (though it does not crash). Looking around for what might cause this, it seems like the most common is some overwriting of shared memory (e.g. opening a file without specifying only rank 0). Is this right? What else might cause what seems to be inter-node communication problems? For what it's worth, it is just this library; we run plenty of other MPI programs on the same cluster with the same compilers without this issue. I find it very odd, since as far as I've seen you never explicitly handle inter-node communication in MPI, just between different processes.
My guess is there is a bug in the code; what kind of functions/common pitfalls with MPI would lead to this kind of behaviour? I haven't seen this kind of issue before (and was unaware it was even possible) and would like to know how to debug it, preferably without hogging several nodes from a shared cluster. I also want to write unit tests that can prevent this sort of thing from happening again, but I have no idea how to write tests for things that only appear on several nodes. It would also be especially helpful if someone could share/link to a minimal program that exhibits this behaviour.
It's a large complicated codebase that I don't fully understand, but it is written modularly. I am interested in trying to write unit tests for it to pinpoint the issue (not to mention improve maintainability). How can I test for an error that only shows up with several nodes? I know pFUnit has MPI support and it looks like they have a mock MPI which you can run quickly to "pretend" you're in parallel without bothering your cluster. However, it looks like I can only mock the number of processes and not the number of nodes. Can I do this with pFUnit? Otherwise, is there some other unit testing framework that would do the trick? Currently there are a few tests written with FRUIT but this looks like a rather minimal framework for testing MPI programs.
Edit: I ran some more tests to make sure it wasn't just the larger number of processes. Indeed, it is the number of nodes changing the results. I ran the script for 20 processes distributed among 1,2,4 nodes and the results were all slightly different. I reran them a few times and they are consistent.
Edit 2: The code uses shared memory. While I do not understand how, there are comments labelling variables like
! MPI shared memory window. There is also a small snippet of code that uses openMP
!$omp parallel do with a few shared variables. However, commenting out the openMP directives gives the same behaviour.