# How to to easily reproduce published results in my own articles using my own code

I wrote a program/library which I used to obtain results in an article. (Here it is, but my question is general.) I have tests that I run regularly using ctest (it takes a few minutes to run). In order to reproduce some tables or figures in the article, I have to construct a script or a simple driver program, that runs maybe 10 minutes, sometimes more, so I don't want to have this part of the regular test suite. At the same time, I want to make sure that the results from the article can be:

• reproduced later
• make sure they still give the same/correct results after I keep developing the library

Currently I try to have a small driver program that I run as part of the regular test suite, and if I want to reproduce results from the article, I uncomment some lines in there. Of course, I never know which exact lines and if I have to tweak some other parameters in order to get precisely the same results as in the article.

I also tried to have a Python script that calculates the exact figures/tables from the article. Such a script typically stops working after an update to the library, because it is not being run on a regular basis (takes too much time).

The best method that occurred to me is to have a Fortran (or C/C++) example, that will be regularly compiled (as part of the library), but not run in regular test suite. That way, at least I know that it compiles fine (and thus hopefully also runs). And I'll test some simple (smaller) example as part of a regular test suite.

What are optimal ways to handle this problem?

-
Great question, my first reaction is that you should divide your tests into quick regressions that can be run swiftly and are performed before every commit and longer regressions that you would want to use as part of a continuous integration effort. Are you specifically in the situation where you only have tests of the former variety and have not divided them yet? – Aron Ahmadia Dec 23 '12 at 21:38
I have lot of tests that run quickly, see here: github.com/certik/hfsolver/tree/master/src/tests, but I don't know how to handle the actual calculations for the article (e.g. 10 min for each table/figure gives easily couple hours total). – Ondřej Čertík Dec 23 '12 at 21:41
Run the long tests automatically nightly (or weekly, monthly, etc.) using a continuous integration server. Since you don't have to pay attention to anything but the results, you won't care how long they take. – David Ketcheson Dec 28 '12 at 20:05

Do you have any parallel tests that can only run on supercomputers, for example, any large-scale tests against p4est? – Aron Ahmadia Jan 2 '13 at 22:03