19
$\begingroup$

Reproducible research in computation aims to make the code needed to generate the results in a computational paper available to other researchers so that they can run this code to reproduce the results in that paper. I'd like to make all of my research reproducible, but I'm running into a bit of a snag: a couple papers I am working on use an in-house automatic differentiation package (called DAEPACK) on a proprietary library (CHEMKIN-II; license terms unclear).

It would be overly time-consuming to replace these software components with open-source versions. An open-source replacement for CHEMKIN-II exists called Cantera, but Cantera is in C++, whereas CHEMKIN-II is in Fortran 77. It would require a lot of effort to modify enough of the Cantera code so that it could be processed by automatic differentiation tools for C++.

Given that I need these proprietary packages, what is the best way to make my research as reproducible as possible, assuming that researchers may not have access to CHEMKIN-II? Since DAEPACK is a source-to-source translator, I don't necessarily need to distribute DAEPACK; I might be able to include its output, which would be Fortran source files that calculate derivatives.

More generally, if you require proprietary software in your work, and that proprietary software isn't widely available (i.e, isn't MATLAB, Mathematica, etc.), how do you make your work reproducible?

| cite | improve this question | |
$\endgroup$
  • $\begingroup$ Does DAEPACK modify the CHEMKIN-II source code, or does it just use CHEMKIN-II internally? $\endgroup$ – Dan Jan 14 '12 at 0:32
  • $\begingroup$ DAEPACK parses the CHEMKIN-II source code and generates Fortran source files that contain subroutines that calculate Jacobian matrices, sparsity patterns, and interval extensions. It doesn't modify the source code; it's like a compiler in that sense. $\endgroup$ – Geoff Oxberry Jan 14 '12 at 0:40
  • $\begingroup$ If the output of DAEPACK is modified CHEMKIN-II source, why are you allowed to distribute it? $\endgroup$ – Dan Jan 14 '12 at 0:43
  • $\begingroup$ It's derived work, but it's not clear to me if the source is modified. Variable names are copied, but by that argument, many of the programs I've written are "modified CHEMKIN-II source". I'd have to check that point. $\endgroup$ – Geoff Oxberry Jan 14 '12 at 0:52
  • $\begingroup$ Does the CHEMKIN-II license allow you to distribute unmodified source freely, and restrict how you distribute changes? I can't find a copy of it's license online. $\endgroup$ – Dan Jan 14 '12 at 0:57
13
$\begingroup$

I don't think that the definition of "reproducible research" requires that the author provide for free all of the tools that are needed to reproduce the results obtained. If some of it is proprietary, then it is incumbent on the future user, not the author, to make the arrangements to acquire the needed software. (You wouldn't expect to have to build the software for somebody else to reproduce your results, right?) You would have the responsibility, of course, of specifying exactly which release of the proprietary software you used, but it is unreasonable to expect you to make a research code compatible with open-source software just because it is open-source.

[As an aside, in addition to talking about the reproducibility of software when some of it is proprietary, I think a more fundamental question needs to be addressed first: does the software give reproducible results across different installations (for instance, for different architectures and compiler versions? For instance, roundoff errors and differences in allocations as a result of differing numbers of processors means that many molecular simulation calculations will give different results just by changing the number of processors included in the allocation. (The statistical averages should reach the same place, but the trajectories may and probably will end up very different.)]

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ I certainly agree that reproducible research does not oblige the author to provide all required software for free. However, software that is not publicly available at all does pose a problem. One of the goals of the reproducible research movement is to help build an awareness of this problem in the community, encouraging people to make research code public even if they don't plan to provide any support for it, just for the sake of reproducibility. $\endgroup$ – khinsen Jan 15 '12 at 10:29
  • 2
    $\begingroup$ Code sold commercially is still available code. If it is no longer possible to obtain a license, that is of course a different matter. But the original author of a code shouldn't have to rewrite a code to accommodate another library only for reproducibility purposes. (There's an argument to be made for greater code viability and usage, but the reproducibility argument is not prima facie necessary--although it could be argued as sufficient.) $\endgroup$ – aeismail Jan 15 '12 at 10:47
4
$\begingroup$

I have never been of the opinion that "reproducible research" means that your fellow readers and scientists should only have to click "Run" (or compile or what have you) to entirely reproduce your results. If nothing else, that would kill the research of anyone who uses a proprietary statistical analysis package, or make things "unreproducable" if they were made for a platform or language that fell out of fashion. Or for that matter, is someone wants to reproduce your code, but doesn't understand/use the language you use.

CrossValidated had a similar discussion on the topic - does "reproducible" mean you can replicate my analysis on my data, or that the experiment itself can be rerun from scratch and provide confirmatory results. The link is here: https://stats.stackexchange.com/questions/14999/how-are-we-defining-reproducible-research

I think the more important element is to describe what you did on the level where it could be recoded by someone who was sufficiently interested, and if commercial tools were used, those tools are mentioned. Because that serves both the interests of replicating the experiment and double-checking for code errors.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

In your case, where you may not even have rights to distribute any of the code you used at all, a very in-depth description of the algorithm might be the best you can do. Depending on how complex your code is, someone might be able to reproduce whatever is of interest to them from whatever tools they have available.

Otherwise, ask around and see how much of your work you're allowed to release, and then release that.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ A lot of this comes from this chat conversation. $\endgroup$ – Dan Jan 14 '12 at 2:47
1
$\begingroup$

The best way to do reproducible research is not to rely on anything short lived like specific codes on specific platform. Yesterday we were computing on punched cards, today on electrons in silicon, tomorrow it may be on lasers with proteins or even living cells, the day after tomorrow on quantum dots, and who knows what will be next week.

Reproducible research should make its concepts and ideas clear and not promote the currently available tools like hardware or software to a prerequisite to reproduce the research results. Algorithms used should be detailed in a clear universal form; current version of a specific hardware, compiler or software package can only be used as a tool to demonstrate the research results.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I think we need both - the concepts and ideas for human understanding and long-term interpretability, and the code for short-term verification. $\endgroup$ – khinsen Jan 15 '12 at 10:31
1
$\begingroup$

Publishing the generated Fortran code sounds like a very good solution in your case.

As a general rule, I think that the use of unpublished software is incompatible with the goals of reproducible research. That's one of the reasons why I see reproducible research as a long-term goal rather than something one could require to be applied immediately.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

I work for Elsevier. My company has started using the Collage framework (developed in response to the Executable Paper Grand Challenge) in journal issues to enable authors to publish chunks of executable code with their article. This feature makes it easier for readers to reproduce results reported in the article and to reuse published material for their own research. Collage supports a wide variety of proprietary software, in addition to open-source software; more information can be found in the informational video here and at the Collage Authoring Environment Website.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.