How do I make sure that the results of my simulations and the results in my paper are always in sync?

Question

In one of my papers, I list some numerical results in addition to some figures. What I'd like to do is make sure that the numerical results in my paper always agree with the code. Right now, I just directly copy the numerical results from my simulation output into the paper, which is extremely simple and low-tech, but error-prone, because I could mis-copy the results, or forget to sync the paper results with the code output.

Is there a good way to keep the numerical results I quote in my papers in sync with the results generated by my code? (Here, I assume that executing the code is easy and practical to do whenever I want to update my paper.) These numerical results don't necessarily lend themselves to tabular form. Sometimes, I have tables in manuscripts, but more commonly, I have simulation parameters listed as numbers in equations. An example would be something like:

\begin{align} \mathbf{y}^{*} = (y_{1}^{*}, \ldots, y_{n}^{*}) \end{align}

where I'd like to replace the elements of the initial condition $\mathbf{y}^{*}$ with the actual parameters I use in a simulation that numerically integrates an system of ordinary differential equations. Using a table for one-off data such as this example seems like overkill and more ink than necessary.

I presume that figures are an easier case: whenever the document is "built" (from LaTeX source, Markdown, RST, etc.), start the build process by executing the code. However, if people have better suggestions for keeping figures generated by my simulations in sync with my paper, I'd love to hear them.

Answer 1

As some comments have suggested, this approach has long been developed in the R community by building on Sweave and more recently, knitr. Obviously this approach has the disadvantage of being language-specific at the moment, but the advantage that its regularly used in academic papers.

Use of Sweave in real publications

• The Journal of Biostatistics encourages these submissions, and places a kitemark letter "R" on academic papers in which the reproduciblity editor has been able to run the code and data and get these results.
• Essentially all publications in the R-Journal are built on Sweave.

Of course more commonly Sweave/knitr users submit just the final output to most journals, with the personal confidence that the methods indeed reproduce the results. Likewise sweave is often used to create slides for professional talks, etc.

Useful features

To be useful in practice, such a system should have certain features. With a critical mass of users, many of these are well-developed in knitr. A few highlights:

• Caching. Writing in a format such as markdown or latex that must be compiled to see the output makes this including the code impossible when results cannot be cached. Clever caching in knitr actually makes debugging intensive code much easier than working in pure R, since it is not necessary to re-run successful chunks.

• Toggle code display. In a formal publication one may not want any of the underlying code to be visible on the output. Meanwhile it is often valuable to show the (nicely formatted, syntax-highlighted) code that produces the results in the output format when you want the reader to see exactly what you are typing.

• Community. Perhaps the most compelling advantage of this approach over a homebrewed solution is the number of folks familiar with the model and trying to improve it.

• Lots of nice examples of other features can be found on the knitr webpage demos, including tools for "lightweight publishing" or sharing on the web. Many of these features help make this approach more compatible with one's general workflow, rather than something to do only when writing the manuscript.

Historical footnote.

While rooted in Knuth's "literate programming", as the name suggests, the emphasis is quite different, as literate programming focuses on connecting software code and documentation (in the R community that role is played by Roxygen, which traces its roots to a different branch of the "literate programming tree").

Going beyond

In principle we might ask much more from a dynamic document, such as the ability of the reader to change the inputs and see the outputs without having to edit and recompile the entire document, for instance, using an interactive online platform. XDynDocs may be a step in this direction.

Other approaches

• Might want to keep an eye on dexy.it
• In a slightly different approach: A growing number of economics papers host code associated with publications on http://www.runmycode.org/, which will re-run the analyses on the cloud, and allow custom input parameters or custom input data.

Answer 2

What you are asking for is the Elsivier grand challenge of the "Executable Paper". While many approaches have been tried, none are as compelling as the authors might suggest. Here are a few examples of techniques used.

Madagascar Project takes your approach, inside the make script have the simulations run that produce the figures and paper simultaneously.

IPython Notebook provides a document that one can execute as you read and produce figures to your hearts content. (I've seen word plugins, Mathematica, and numerous other solutions used in the same fashion)

VisTrails uses a service oriented architecture approach and provides a "providence" or "workflow" manager. Basically you register hooks to code then design a work flow or experiment that reproduces your work. It has been used on many types of codes, even HPC clusters. With this approach you will have a way to replay the experiments.

There are tons of these type solutions out there, but those are three I was impressed with. Its a hard problem and one I believe we really aren't even close to addressing. We can't even get people to release their code with their papers, how can we expect them to reproduce the results =P

Answer 3

I have not had a lot of success in using other people's solutions to this problem. I usually just want something simple that works for me and gets the job done. To this end, I generally try to write one python script which is in charge of running all the results, parsing the output, as well as building the figures/tables.

I write my codes to generate datafiles which contain the results in some text format. You can avoid rerunning these results in your script by first testing for the existence of the output file (in python using os.path.isfile() for example). If you want to rerun your results, simply remove the datafiles. If the datafiles exist, then I run a parser of these files. For this, the python module for regular expressions is very useful (re).

Then from the parsed output I create the figures or tables. For tables in latex you can write the code to generate the table into a separate file (I use a .tbl extension) and then include this into your latex file. They key for me is to use 1 python script. If I have many, then I am later wondering which one is which and what they do. If this description is too vague I can send you some examples.

Answer 4

Even more important, in my opinion, is making sure that you can figure out how to re-generate all your results from scratch in a month or a year (for instance, when referees ask you to add or modify something). For that purpose, what I do is include a text file with very detailed directions on how to reproduce all results. It's best if you test these out by having someone else (like a co-author) try them. I recommend that you also provide these instructions (and all your code) to referees and readers.

Here is an example (actually prepared by my co-author, Aron Ahmadia).

Answer 5

Emacs's orgmode in combination with Babel achieves that. Babel can execute code snippets from various programming and scripting languages, e.g., it could open the file containing the simulation data and put it into a table in orgmode, which can be exported to LaTeX (and many other formats). It takes quite a while getting used to all the key combos in orgmode, but once it's running everything is automatic.

Answer 6

If running all your code is cheap then you could do something low-tech like the following:

You could template your documents with formatted strings so that they look like this

"we observed a %(fractional_improvement)s increase in ..."

Have python scripts that look like this

results = {"alpha"                  : run_alpha_computation(...),
           "fractional_improvement" : run_fi_computation(...), 
           ...}

And then do something like this

for fn in filenames:
    file = open(fn);      s = file.read();       file.close()
    file = open(fn, 'w'); file.write(s%results); file.close()

You could then wrap this up in a Makefile.

Answer 7

If you're using LaTeX, a relatively low-tech solution is to have your code spit out a file (or else use a script to filter from your code's output) containing a whole bunch of lines like this:

\newcommand{\myresults1}{<value>}

Then you can use the \input command to add that file into your document, and use the defined commands to place the values.

Answer 8

I work for Elsevier. My company has started using the Collage framework (developed in response to the Executable Paper Grand Challenge) in journal issues to enable authors to publish chunks of executable code with their article. This feature makes it easier for readers to reproduce results reported in the article and to reuse published material for their own research. Collage supports a wide variety of open-source and proprietary software; more information can be found in the informational video here and at the Collage Authoring Environment Website.