1
$\begingroup$

We've recently ported our Python/Fortran simulation code to a new supercomputer. Some (not all) of the tests (simulations) that we've run on the new platform yield results that are significantly different from those we got on the "old" cluster. We see ${\cal O}(1)$ numerical differences between output variables on the two respective systems, much larger than what we'd expect from just different hardware platforms and compilers. We've checked the usual suspects (I/O files, simulation settings, ...) but found no obvious culprit.

Hence my question: are there (Python) tools that may help us with the debug process? Something that e.g. checks output directories and recursively compares output folders (ideally going into netCDFs, GRIBs, etc. to compare variable values), comes up with diagnostic plots, flags discrepancies, etc?

Improve this question
$\endgroup$
6
  • 3
    $\begingroup$ Does your simulation involve generation and use of random numbers? $\endgroup$
    – Mark L. Stone
    Sep 6, 2019 at 17:05
  • $\begingroup$ @MarkL.Stone yes, but the random seeds are the same on both platforms $\endgroup$
    – GoHokies
    Sep 6, 2019 at 17:15
  • 2
    $\begingroup$ So are the random number generators themselves identical? $\endgroup$
    – Victor Eijkhout
    Sep 6, 2019 at 18:09
  • 1
    $\begingroup$ @GoHokies: But the PRNG may be different. C++, for instance, makes no guarantees about how the PRNG is implemented. I'd recommend starting by adding a simple unittest for this. If it's the problem, that'll save you the time of coming up with a full fledged comparison solution. $\endgroup$
    – Richard
    Sep 6, 2019 at 18:11
  • $\begingroup$ @VictorEijkhout it's only the Python code that generates random numbers. the conda environment is "identical" on both platforms (same py version, numpy lib, etc.), so I'm expecting the RNGs to be identical. am I wrong in expecting that? $\endgroup$
    – GoHokies
    Sep 6, 2019 at 18:57

2 Answers 2

Reset to default
4
$\begingroup$

How identical are your two runs? Same number of processors at least?

If you're doing a parallel reduction then one platform can use a different algorithm, so because of lack of associativity you get different results. After that, if your computation is not intrinsically stable the error amplifies. Which of course also means that on either platform the result is meaningless, but we'll leave that aside for the moment.

Improve this answer
$\endgroup$
2
  • $\begingroup$ yes, the same number of processors for both jobs. a numerical instability is, in my opinion, unlikely - the code has been operational for a while and it has all kinds of bells and whistles that should guard against (or at the very least flag) numerical instabilities. nevertheless, I take your point. I guess I'll have to manually track down the first occurrence of non-negligible numerical differences, and start debugging from there. $\endgroup$
    – GoHokies
    Sep 6, 2019 at 18:54
  • 1
    $\begingroup$ Without knowing what your computation is I'm not ruling out numerical problems. $\endgroup$
    – Victor Eijkhout
    Sep 7, 2019 at 21:18
4
$\begingroup$

Have you considered the possibility that they're both wrong, or both "right"?

Do a quick backward error computation $\left\| L u - f\right\|$ (with requisite adjustments for you case) at the end of both simulations and then you'll be able to say whether one or the other is right, or if both are perfectly reasonable solutions, but just exhibit psychologically distressing differences in the forward error $\left\|u_1 - u_2\right\|$.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.