24
$\begingroup$

Reproducibility has become more and more important in computational science research. (For instance, see this article by Roger Peng in Science; I'm aware of other such articles and web sites also.) However, it's not clear to me how much information I should include with a journal article (or online) to make my computational research reproducible (assuming there are no other obstacles, like intellectual property agreements). Are there any guidelines out there, and if not, could people suggest what steps researchers should take to make their computational science research reproducible?

Of particular use in answers would be possible ways to implement those suggestions -- workflows, basically. Workflows that are system-agnostic or Linux-based are preferable. Also, discussing any relevant personal experience you've had would also be helpful.

In my particular case, I'm writing a theoretical paper with a couple example calculations that are simple enough that they can be done in MATLAB. I'd think that in this case, including the MATLAB script, as well as noting the specific version of MATLAB on my machine, would be enough to ensure reproducibility. However, I'm certain there are more complicated scenarios out there, and advice on how to carry out reproducible research would be very helpful to know for future projects.

Improve this question
$\endgroup$

5 Answers 5

Reset to default
18
$\begingroup$

In rough order of importance.

Source Code

  1. Make the code that implements the key aspects of your algorithm available. Even if the user can't build or run it, they can read exactly what is done. I have several times noticed simple decisions that weren't documented in a paper, but which a couple minutes with the source code answered conclusively.
  2. Make it runnable. This involves documenting the versions of dependent libraries and usually requires you to write somewhat portable code. Make sure it builds on at least one machine other than your own (it's easy to have hidden dependencies if you have never built it in a clean environment).
  3. Specify the version of the code that was used. If not a formally released version (and sometimes even then), document the SHA1 of the version. (This most naturally applies to DSCMs like Git and Mercurial, but can be used anywhere.) This is a very reliable way to guarantee that someone really has the same version of the code.
  4. Include configuration and host parameters including compiler vendor, versions, and optimization flags, system libraries like libc, CPU type, and memory type and topology (especially for performance studies).

Run-time parameters/input files

Include the complete input specification. If it was generated by a script, include that script. If it is huge data, document how to obtain and process the data. If your algorithm has randomness, specify the random number generator and seed that was used.

Scripts to generate figures and tables

It is very helpful to include these scripts, both to clarify any questions about what the figures really show and to let the reader experiment with how things change if they change parameters or modify the algorithm.

Improve this answer
$\endgroup$
2
  • $\begingroup$ How important would it be to include unit tests? How well should I document code I include for reproducibility? $\endgroup$
    – Geoff Oxberry
    Commented Apr 7, 2012 at 6:24
  • $\begingroup$ If you just want reproducibility of results, unit tests and manual pages/user's manuals are not necessary. If you are trying to attract future coauthors or users of your software (citations...), then writing the software for reuse and documenting it thoroughly is worthwhile. Note that regardless of whether your software is intended for others to use, tests and documentation may well save you time in the long run, just because it lets you move around and experiment more confidently. $\endgroup$
    – Jed Brown
    Commented Apr 7, 2012 at 7:12
5
$\begingroup$

Most journals aren't set up for this in any formal way, but we've recently founded the Archive of Numerical Software that is specifically intended to have the source code and everything else that's necessary be part of the article. Check it out: http://journals.tdl.org/ans Submissions are welcome!

Improve this answer
$\endgroup$
6
  • 2
    $\begingroup$ Assuming you use the pet software project of some member of the editorial board. I can't help but feel that this requirement degrades the integrity of the journal. $\endgroup$
    – Jack Poulson
    Commented Apr 10, 2012 at 16:55
  • 1
    $\begingroup$ @JackPoulson: This is a point that we've discussed at length among the editors, and which we have discussed at even greater length with others in the community. I think we all see your point, but at the same time, we felt like we can't do it any other way for two reasons: (i) We don't know where to find reviewers for random project X. (ii) There is a certain consensus in the community of which projects are high quality and which are not; we didn't want ANS to become an outlet for every wannabe project. As we state on the webpage, we would like to include all high-quality packages eventually. $\endgroup$
    – Wolfgang Bangerth
    Commented Apr 10, 2012 at 20:17
  • 2
    $\begingroup$ For what reason does a reviewer have to be familiar with "random project X" instead of "random field X"? I bring this up because I think the journal is missing out on a significant class of papers, as there are severe limitations on what is possible to implement with the given list of packages. For instance, there can be no fundamental contributions to parallel computing, as anything involving dipping into calls to MPI, or, heaven forbid, BLAS or LAPACK, would seem to violate the standards of the journal. I would be happy to talk about this further offline. $\endgroup$
    – Jack Poulson
    Commented Apr 10, 2012 at 20:44
  • $\begingroup$ @JackPoulson: As far as I can tell, if you want to make a fundamental contribution to parallel computing, in addition to writing a paper in another journal, you can also write a "Library Introduction" paper in ANS to ensure that your library is added to the approved list of libraries. $\endgroup$
    – Geoff Oxberry
    Commented Apr 11, 2012 at 20:04
  • 1
    $\begingroup$ @GeoffOxberry: Wolfgang and I had an amicable conversation, and the common ground conclusion was that the list of linear algebra libraries should be greatly expanded, but that the goal of the journal is for "high-quality" libraries rather than simply reproducibility, and therefore there must be some vetting process. $\endgroup$
    – Jack Poulson
    Commented Apr 11, 2012 at 20:10
3
$\begingroup$

In

Stodden, V. 2009. “The Legal Framework for Reproducible Scientific Research.” CiSE.

Victoria Stodden recommends publishing the full "research compendium", and lists the following components on p. 38:

  1. The research paper
  2. The data - including documentation and code for processing the data
  3. The experiment — all source code; documentation, parameters, settings, and operating system dependencies
  4. The results of the experiment — figures, data, illustration source files; and documentation and explanation of the processing of the experimental results
  5. Any auxiliary material
Improve this answer
$\endgroup$
2
$\begingroup$

At least, the Source code and data you used to perform your experiments should be accesible somewhere. Add instructions to build your code if necessary. Really there are so few open access journals that there is no an open and established rule.

Improve this answer
$\endgroup$
2
$\begingroup$

I work for Elsevier. My company has started using the Collage framework (developed in response to the Executable Paper Grand Challenge) in journal issues to enable authors to include all of the data and code needed to reproduce the results and figures in their papers. This feature makes it easier for readers to reproduce results reported in the article and to reuse published material for their own research. Collage supports a wide variety of open-source and proprietary software; more information can be found in the informational video here and at the Collage Authoring Environment Website.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.