1
$\begingroup$

We've recently ported our Python/Fortran simulation code to a new supercomputer. Some (not all) of the tests (simulations) that we've run on the new platform yield results that are significantly different from those we got on the "old" cluster. We see ${\cal O}(1)$ numerical differences between output variables on the two respective systems, much larger than what we'd expect from just different hardware platforms and compilers. We've checked the usual suspects (I/O files, simulation settings, ...) but found no obvious culprit.

Hence my question: are there (Python) tools that may help us with the debug process? Something that e.g. checks output directories and recursively compares output folders (ideally going into netCDFs, GRIBs, etc. to compare variable values), comes up with diagnostic plots, flags discrepancies, etc?

$\endgroup$
  • 3
    $\begingroup$ Does your simulation involve generation and use of random numbers? $\endgroup$ – Mark L. Stone Sep 6 at 17:05
  • $\begingroup$ @MarkL.Stone yes, but the random seeds are the same on both platforms $\endgroup$ – GoHokies Sep 6 at 17:15
  • 2
    $\begingroup$ So are the random number generators themselves identical? $\endgroup$ – Victor Eijkhout Sep 6 at 18:09
  • 1
    $\begingroup$ @GoHokies: But the PRNG may be different. C++, for instance, makes no guarantees about how the PRNG is implemented. I'd recommend starting by adding a simple unittest for this. If it's the problem, that'll save you the time of coming up with a full fledged comparison solution. $\endgroup$ – Richard Sep 6 at 18:11
  • $\begingroup$ @VictorEijkhout it's only the Python code that generates random numbers. the conda environment is "identical" on both platforms (same py version, numpy lib, etc.), so I'm expecting the RNGs to be identical. am I wrong in expecting that? $\endgroup$ – GoHokies Sep 6 at 18:57
4
$\begingroup$

How identical are your two runs? Same number of processors at least?

If you're doing a parallel reduction then one platform can use a different algorithm, so because of lack of associativity you get different results. After that, if your computation is not intrinsically stable the error amplifies. Which of course also means that on either platform the result is meaningless, but we'll leave that aside for the moment.

$\endgroup$
  • $\begingroup$ yes, the same number of processors for both jobs. a numerical instability is, in my opinion, unlikely - the code has been operational for a while and it has all kinds of bells and whistles that should guard against (or at the very least flag) numerical instabilities. nevertheless, I take your point. I guess I'll have to manually track down the first occurrence of non-negligible numerical differences, and start debugging from there. $\endgroup$ – GoHokies Sep 6 at 18:54
  • 1
    $\begingroup$ Without knowing what your computation is I'm not ruling out numerical problems. $\endgroup$ – Victor Eijkhout Sep 7 at 21:18
4
$\begingroup$

Have you considered the possibility that they're both wrong, or both "right"?

Do a quick backward error computation $\left\| L u - f\right\|$ (with requisite adjustments for you case) at the end of both simulations and then you'll be able to say whether one or the other is right, or if both are perfectly reasonable solutions, but just exhibit psychologically distressing differences in the forward error $\left\|u_1 - u_2\right\|$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.