Physics, biology, chemistry etc. have different sets of rules for making experiments: what events are considered relevant, how to avoid contamination of samples, how to create and fix a reproducing process and so on.
What are standards, protocols and best practices in ensure accuracy and reproducibility in numerical experiments?
A lot has been written about how to design, execute, and report the results of computational experiments. This has obvious connections with open source software and the broader "open science" movement. Another important issue is the difference between "my code is faster than yours" research and research that helps us to better understand the properties of algorithms.
Some references on computational experiments and testing that you might want to look at include:
Coffin, Marie, and Matthew J. Saltzman. 2000. “Statistical Analysis of Computational Tests of Algorithms and Heuristics.” INFORMS Journal on Computing 12 (1): 24–44.
Crowder, Harlan, Ron S. Dembo, and John M. Mulvey. 1979. “On Reporting Computational Experiments with Mathematical Software.” ACM Transactions on Mathematical Software (TOMS) 5 (2): 193–203.
Feitelson, Dror G. 2006. “Experimental Computer Science: The Need for a Cultural Change.”
Hooker, John N. 1995. “Testing Heuristics: We Have It All Wrong.” Journal of Heuristics 1 (1): 33–42.
McGeoch, Catherine C. 2012. A Guide to Experimental Algorithmics. Cambridge University Press.
