# simulation outputs differ across hardware platforms

We've recently ported our Python/Fortran simulation code to a new supercomputer. Some (not all) of the tests (simulations) that we've run on the new platform yield results that are significantly different from those we got on the "old" cluster. We see $${\cal O}(1)$$ numerical differences between output variables on the two respective systems, much larger than what we'd expect from just different hardware platforms and compilers. We've checked the usual suspects (I/O files, simulation settings, ...) but found no obvious culprit.

Hence my question: are there (Python) tools that may help us with the debug process? Something that e.g. checks output directories and recursively compares output folders (ideally going into netCDFs, GRIBs, etc. to compare variable values), comes up with diagnostic plots, flags discrepancies, etc?

• Does your simulation involve generation and use of random numbers? – Mark L. Stone Sep 6 '19 at 17:05
• @MarkL.Stone yes, but the random seeds are the same on both platforms – GoHokies Sep 6 '19 at 17:15
• So are the random number generators themselves identical? – Victor Eijkhout Sep 6 '19 at 18:09
• @GoHokies: But the PRNG may be different. C++, for instance, makes no guarantees about how the PRNG is implemented. I'd recommend starting by adding a simple unittest for this. If it's the problem, that'll save you the time of coming up with a full fledged comparison solution. – Richard Sep 6 '19 at 18:11
• @VictorEijkhout it's only the Python code that generates random numbers. the conda environment is "identical" on both platforms (same py version, numpy lib, etc.), so I'm expecting the RNGs to be identical. am I wrong in expecting that? – GoHokies Sep 6 '19 at 18:57

Do a quick backward error computation $$\left\| L u - f\right\|$$ (with requisite adjustments for you case) at the end of both simulations and then you'll be able to say whether one or the other is right, or if both are perfectly reasonable solutions, but just exhibit psychologically distressing differences in the forward error $$\left\|u_1 - u_2\right\|$$.