Physics, biology, chemistry etc. have different sets of rules for making experiments: what events are considered relevant, how to avoid contamination of samples, how to create and fix a reproducing process and so on.

What are standards, protocols and best practices in ensure accuracy and reproducibility in numerical experiments?

  • 2
    $\begingroup$ I have never come across a set of best practice rules, but based on working with what is being published, I would say that even for reproducability alone there are major issues in present practice: Results may be influenced by hardware platform, operating system (and version), compiler (and version), compilation switches, and application configuration settings. Rarely do I find this documented with a sufficient amount of detail. $\endgroup$
    – njuffa
    Commented Apr 25, 2018 at 21:19

1 Answer 1


A lot has been written about how to design, execute, and report the results of computational experiments. This has obvious connections with open source software and the broader "open science" movement. Another important issue is the difference between "my code is faster than yours" research and research that helps us to better understand the properties of algorithms.

Some references on computational experiments and testing that you might want to look at include:

Coffin, Marie, and Matthew J. Saltzman. 2000. “Statistical Analysis of Computational Tests of Algorithms and Heuristics.” INFORMS Journal on Computing 12 (1): 24–44.

Crowder, Harlan, Ron S. Dembo, and John M. Mulvey. 1979. “On Reporting Computational Experiments with Mathematical Software.” ACM Transactions on Mathematical Software (TOMS) 5 (2): 193–203.

Feitelson, Dror G. 2006. “Experimental Computer Science: The Need for a Cultural Change.”

Hooker, John N. 1995. “Testing Heuristics: We Have It All Wrong.” Journal of Heuristics 1 (1): 33–42.

McGeoch, Catherine C. 2012. A Guide to Experimental Algorithmics. Cambridge University Press.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.