# How do I make sure that the results of my simulations and the results in my paper are always in sync?

In one of my papers, I list some numerical results in addition to some figures. What I'd like to do is make sure that the numerical results in my paper always agree with the code. Right now, I just directly copy the numerical results from my simulation output into the paper, which is extremely simple and low-tech, but error-prone, because I could mis-copy the results, or forget to sync the paper results with the code output.

Is there a good way to keep the numerical results I quote in my papers in sync with the results generated by my code? (Here, I assume that executing the code is easy and practical to do whenever I want to update my paper.) These numerical results don't necessarily lend themselves to tabular form. Sometimes, I have tables in manuscripts, but more commonly, I have simulation parameters listed as numbers in equations. An example would be something like:

\begin{align} \mathbf{y}^{*} = (y_{1}^{*}, \ldots, y_{n}^{*}) \end{align}

where I'd like to replace the elements of the initial condition $\mathbf{y}^{*}$ with the actual parameters I use in a simulation that numerically integrates an system of ordinary differential equations. Using a table for one-off data such as this example seems like overkill and more ink than necessary.

I presume that figures are an easier case: whenever the document is "built" (from LaTeX source, Markdown, RST, etc.), start the build process by executing the code. However, if people have better suggestions for keeping figures generated by my simulations in sync with my paper, I'd love to hear them.

## 8 Answers

As some comments have suggested, this approach has long been developed in the R community by building on Sweave and more recently, knitr. Obviously this approach has the disadvantage of being language-specific at the moment, but the advantage that its regularly used in academic papers.

### Use of Sweave in real publications

• The Journal of Biostatistics encourages these submissions, and places a kitemark letter "R" on academic papers in which the reproduciblity editor has been able to run the code and data and get these results.
• Essentially all publications in the R-Journal are built on Sweave.

Of course more commonly Sweave/knitr users submit just the final output to most journals, with the personal confidence that the methods indeed reproduce the results. Likewise sweave is often used to create slides for professional talks, etc.

### Useful features

To be useful in practice, such a system should have certain features. With a critical mass of users, many of these are well-developed in knitr. A few highlights:

• Caching. Writing in a format such as markdown or latex that must be compiled to see the output makes this including the code impossible when results cannot be cached. Clever caching in knitr actually makes debugging intensive code much easier than working in pure R, since it is not necessary to re-run successful chunks.

• Toggle code display. In a formal publication one may not want any of the underlying code to be visible on the output. Meanwhile it is often valuable to show the (nicely formatted, syntax-highlighted) code that produces the results in the output format when you want the reader to see exactly what you are typing.

• Community. Perhaps the most compelling advantage of this approach over a homebrewed solution is the number of folks familiar with the model and trying to improve it.

• Lots of nice examples of other features can be found on the knitr webpage demos, including tools for "lightweight publishing" or sharing on the web. Many of these features help make this approach more compatible with one's general workflow, rather than something to do only when writing the manuscript.

### Historical footnote.

While rooted in Knuth's "literate programming", as the name suggests, the emphasis is quite different, as literate programming focuses on connecting software code and documentation (in the R community that role is played by Roxygen, which traces its roots to a different branch of the "literate programming tree").

### Going beyond

In principle we might ask much more from a dynamic document, such as the ability of the reader to change the inputs and see the outputs without having to edit and recompile the entire document, for instance, using an interactive online platform. XDynDocs may be a step in this direction.

### Other approaches

• Might want to keep an eye on dexy.it
• In a slightly different approach: A growing number of economics papers host code associated with publications on http://www.runmycode.org/, which will re-run the analyses on the cloud, and allow custom input parameters or custom input data.
• Carl, I've been following your website for a while, and your approach with knitr was one of the inspirations for my question. I've also been following dexy for a while because Zed Shaw uses it to build the source for his book How To Learn Python the Hard Way (see the git repo). What I like about dexy as opposed to other literate programming approaches is that code and text are loosely coupled, enabling sane use of a debugger. – Geoff Oxberry Jun 20 '12 at 20:22
• Geoff, cool, thanks for the comments! Anna Nelson is a big believer in decoupling too. R users can accomplish the decoupling in knitr with code externalization, or the new spin() function. Personally I think Greg Wilson's complaints about literate programming are pretty dated though. I had the same terrible experience he describes with noweb, but that convoluted debugging just doesn't exist in the modern tools. For literate C I use doxygen. Knitr is if anything easier to debug than R thanks to caching and environment handling. – cboettig Jun 20 '12 at 22:04

What you are asking for is the Elsivier grand challenge of the "Executable Paper". While many approaches have been tried, none are as compelling as the authors might suggest. Here are a few examples of techniques used.

Madagascar Project takes your approach, inside the make script have the simulations run that produce the figures and paper simultaneously.

IPython Notebook provides a document that one can execute as you read and produce figures to your hearts content. (I've seen word plugins, Mathematica, and numerous other solutions used in the same fashion)

VisTrails uses a service oriented architecture approach and provides a "providence" or "workflow" manager. Basically you register hooks to code then design a work flow or experiment that reproduces your work. It has been used on many types of codes, even HPC clusters. With this approach you will have a way to replay the experiments.

There are tons of these type solutions out there, but those are three I was impressed with. Its a hard problem and one I believe we really aren't even close to addressing. We can't even get people to release their code with their papers, how can we expect them to reproduce the results =P

• In a similar vein there is sweave, which is not something that I use but is intriguing in concept. – dmckee Jun 14 '12 at 22:00
• The Madagascar Project seemed like it could be interesting when I listened to a talk by one of its authors. I have not actually tried to use it though. – Ken Jun 15 '12 at 7:26
• @dmckee: I know people who have had good success with sweave and knitr. I'm leery of literate programming approaches for the same reasons that Greg Wilson gives on Software Carpentry: the paper and the code are too tightly coupled, which makes it hard to run a debugger on the code (and could get in the way of proofing the text). – Geoff Oxberry Jun 15 '12 at 8:49
• I use Sweave for this purpose, it works very nicely and is cOmpatible with Lyx. Org-mode is even better and supports most common languages. – David LeBauer Jun 23 '12 at 3:22

I have not had a lot of success in using other people's solutions to this problem. I usually just want something simple that works for me and gets the job done. To this end, I generally try to write one python script which is in charge of running all the results, parsing the output, as well as building the figures/tables.

I write my codes to generate datafiles which contain the results in some text format. You can avoid rerunning these results in your script by first testing for the existence of the output file (in python using os.path.isfile() for example). If you want to rerun your results, simply remove the datafiles. If the datafiles exist, then I run a parser of these files. For this, the python module for regular expressions is very useful (re).

Then from the parsed output I create the figures or tables. For tables in latex you can write the code to generate the table into a separate file (I use a .tbl extension) and then include this into your latex file. They key for me is to use 1 python script. If I have many, then I am later wondering which one is which and what they do. If this description is too vague I can send you some examples.

• I do this sort of thing for figures already. However, in the papers I'm writing, tables would be an unnatural format in which to present the data. Often times, I really just want to include the initial condition to an ODE (so really, something like 4-6 numbers, spaced by commas), or a whole matrix of numbers as part of the right-hand side of an equation. I like your idea for tables. For those cases I mentioned, I feel that reformatting them as tables would be unnatural, and I'd like to include the data in a more natural format. – Geoff Oxberry Jun 18 '12 at 13:13
• Nathan, would you mind posting examples? I use the same approach, except that I commit the text files into git, and use git to manage the results. I then have Python scripts to produce plots/tables. I have one script per plot or table. – Ondřej Čertík Jun 19 '12 at 22:55
• You can also pipe the output of a shell script directly to latex using the command \input{|"path-to-script.py"}. I think you'd better to put all the parameters in a single python (or whatever language is your favorite) file and use command line parameter to access it like \input{|"path-to-script.py param-name"}. In this case you can include the param file into other scripts to run the simulations. However, it makes the compilation slower and has some other negative points. – Helium Aug 2 '12 at 9:28
• – Helium Aug 2 '12 at 9:28

Even more important, in my opinion, is making sure that you can figure out how to re-generate all your results from scratch in a month or a year (for instance, when referees ask you to add or modify something). For that purpose, what I do is include a text file with very detailed directions on how to reproduce all results. It's best if you test these out by having someone else (like a co-author) try them. I recommend that you also provide these instructions (and all your code) to referees and readers.

Here is an example (actually prepared by my co-author, Aron Ahmadia).

• I've done that before (for my own sanity), and thankfully, it paid off when my adviser asked me to re-generate and re-check results. I've since switched to just dumping the source code of a script that runs everything into an appendix of my drafts so it's there, I know what I did, and I can click one button to get all of the numbers and figures. – Geoff Oxberry Jun 18 '12 at 13:10
• The script doesn't do any installation right now, because it's just a MATLAB script. In the function documentation, it lists dependencies on third-party packages. Those third-party packages, in turn, both have clear documentation on how to install them (and thankfully, are also actively supported, have great developers, and active mailing lists). – Geoff Oxberry Jun 19 '12 at 10:19

Emacs's orgmode in combination with Babel achieves that. Babel can execute code snippets from various programming and scripting languages, e.g., it could open the file containing the simulation data and put it into a table in orgmode, which can be exported to LaTeX (and many other formats). It takes quite a while getting used to all the key combos in orgmode, but once it's running everything is automatic.

• I like org-mode; I use it for outlines. I haven't used it with Babel. I'll have to try it out. – Geoff Oxberry Jun 19 '12 at 21:38
• Here is an excellent overview from Jan 2012 J. Stat. Software jstatsoft.org/v46/i03/paper – David LeBauer Jun 23 '12 at 3:27
• I wrote a tutorial that shows how to convert the LaTeX template from the European Physical Journal A (EPJ A) into an org-mode file. – Melioratus Apr 8 at 15:49

If running all your code is cheap then you could do something low-tech like the following:

You could template your documents with formatted strings so that they look like this

"we observed a %(fractional_improvement)s increase in ..."


Have python scripts that look like this

results = {"alpha"                  : run_alpha_computation(...),
"fractional_improvement" : run_fi_computation(...),
...}


And then do something like this

for fn in filenames:
file = open(fn);      s = file.read();       file.close()
file = open(fn, 'w'); file.write(s%results); file.close()


You could then wrap this up in a Makefile.

• My first thought when I wrote this question was a solution sort of like the one you've proposed. I was originally thinking of something lower-tech like using a macro preprocessor, but Python is probably a better (and certainly more readable) approach, then a build system can handle incremental regeneration of results. – Geoff Oxberry Jun 20 '12 at 3:24
• Really this is just a very basic implementation of something like python server pages. The idea of auto-generated content has been around in the web community for a while. It would be nice to see it migrate over to academia. – MRocklin Jun 20 '12 at 17:13
• Agreed. Jinja2 could be used to do what you've suggested. In fact, that's what dexy does, but with a bunch of cool filters that also handle syntax highlighting and other sundry tasks. – Geoff Oxberry Jun 20 '12 at 23:20

If you're using LaTeX, a relatively low-tech solution is to have your code spit out a file (or else use a script to filter from your code's output) containing a whole bunch of lines like this:

\newcommand{\myresults1}{<value>}

Then you can use the \input command to add that file into your document, and use the defined commands to place the values.

I work for Elsevier. My company has started using the Collage framework (developed in response to the Executable Paper Grand Challenge) in journal issues to enable authors to publish chunks of executable code with their article. This feature makes it easier for readers to reproduce results reported in the article and to reuse published material for their own research. Collage supports a wide variety of open-source and proprietary software; more information can be found in the informational video here and at the Collage Authoring Environment Website.

• The second link is to the wrong thing. – David Ketcheson Jul 25 '12 at 5:05
• @Hylke Koers: Did you mean to put this link: collage.elsevier.com ? – Paul Jul 25 '12 at 23:40
• @Paul: I made the edits; the original second link was to the Collage Google Group. Maybe a better link would be to Collage itself, but my focus was to try to (mostly) preserve the good intentions of the post while removing the parts that made it sound promotional. Feel free to edit the post as you see fit. – Geoff Oxberry Jul 27 '12 at 4:13