4
$\begingroup$

So, my question is if one should ideally keep a record of all seeds that are used when publishing numerical work that involves one or more random number generators (e.g. a stochastic simulation), and if so, whether there's a standard approach to doing this when the number of random number generators depends on the hardware on which the program is run. On the one hand, having an exact record of seeds could be helpful for reproducibility and correctness certification, but on the other, statistical results should be robust enough that the exact seeds shouldn't really matter.

$\endgroup$
3
$\begingroup$

Achieving bit-for-bit reproducibility of results for a code that runs on a variable number of processors in a distributed computing cluster is extremely difficult if not impossible- there are just too many random factors that can change the result. Even on a shared memory multiprocessor, obtaining bit-for-bit reproducibility can be extremely difficult and require significant performance tradeoffs.

For example, Intel's MKL library has a "Conditional Numerical Reproducibility" mode in which the library routines will always return exactly the same results, but this comes at a cost in performance.

You can try rerunning your simulation using the same seeds and the same number of processors to see if you get exactly the same results. Unless your code has been carefully written to ensure reproducibility, it's most likely that you'll see some variations in the results.

If your results aren't bit-for-bit reproducible, then you'll have to decide whether you are willing to dive into the code to resolve these issues or whether you're willing to settle for answers that are not exactly reproducible but are consistently within some tolerance that's acceptable.

In any case, if you're performing a Monte Carlo simulation, you'll need to statistically analyze your output- a single realization of a Monte Carlo simulation with a specified RNG seed might be reproducible, but it's only one realization of many that would result from other seeds.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ I'd add to this that, while right now it would be considered going above and beyond the call to guarantee bit-for-bit reproducibility of your results, some very smart people are developing new tools to help with this like ReproBLAS. $\endgroup$ – Daniel Shapero Oct 24 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.