Take the 2-minute tour ×
Computational Science Stack Exchange is a question and answer site for scientists using computers to solve scientific problems. It's 100% free, no registration required.

I wrote a program/library which I used to obtain results in an article. (Here it is, but my question is general.) I have tests that I run regularly using ctest (it takes a few minutes to run). In order to reproduce some tables or figures in the article, I have to construct a script or a simple driver program, that runs maybe 10 minutes, sometimes more, so I don't want to have this part of the regular test suite. At the same time, I want to make sure that the results from the article can be:

  • reproduced later
  • make sure they still give the same/correct results after I keep developing the library

Currently I try to have a small driver program that I run as part of the regular test suite, and if I want to reproduce results from the article, I uncomment some lines in there. Of course, I never know which exact lines and if I have to tweak some other parameters in order to get precisely the same results as in the article.

I also tried to have a Python script that calculates the exact figures/tables from the article. Such a script typically stops working after an update to the library, because it is not being run on a regular basis (takes too much time).

The best method that occurred to me is to have a Fortran (or C/C++) example, that will be regularly compiled (as part of the library), but not run in regular test suite. That way, at least I know that it compiles fine (and thus hopefully also runs). And I'll test some simple (smaller) example as part of a regular test suite.

What are optimal ways to handle this problem?

share|improve this question
2  
Great question, my first reaction is that you should divide your tests into quick regressions that can be run swiftly and are performed before every commit and longer regressions that you would want to use as part of a continuous integration effort. Are you specifically in the situation where you only have tests of the former variety and have not divided them yet? –  Aron Ahmadia Dec 23 '12 at 21:38
    
I have lot of tests that run quickly, see here: github.com/certik/hfsolver/tree/master/src/tests, but I don't know how to handle the actual calculations for the article (e.g. 10 min for each table/figure gives easily couple hours total). –  Ondřej Čertík Dec 23 '12 at 21:41
2  
Run the long tests automatically nightly (or weekly, monthly, etc.) using a continuous integration server. Since you don't have to pay attention to anything but the results, you won't care how long they take. –  David Ketcheson Dec 28 '12 at 20:05

1 Answer 1

up vote 5 down vote accepted

In deal.II, we have a testsuite that is driven by a regular Unix Makefile. It has a default target that runs all the usual tests, and a separate target for expensive tests. Running each test is done using a generic rule but the default target calls the generic rule only for certain tests and the expensive target calls it for the expensive tests. Because it's all done using a generic rule, it must be up to date at any given time; what may get out of date can only be the lists of names of tests.

share|improve this answer
    
Thanks! Here is the link to the docs: dealii.org/7.2.0/development/testsuite.html#regression_tests and here is the result of the "usual tests": dealii.org/cgi-bin/regression_quick.pl and here of the "expensive tests": dealii.org/cgi-bin/regression.pl, did I get it right? So you run "usual tests" on each revision and "expensive tests" only every couple of revisions? –  Ondřej Čertík Dec 26 '12 at 16:56
    
Not quite. Results for expensive tests are not usually posted to the website. –  Wolfgang Bangerth Dec 26 '12 at 18:34
    
So you run them manually let's say before each release? How long do they take to run? I like your approach. –  Ondřej Čertík Dec 26 '12 at 22:21
1  
Yes, manually. Every once in a while, before releases for sure. For some projects they take an hour or more but because they test only a small part of the library (they mostly test add-on projects to deal.II) it's not always worth or possible running them with every revision. –  Wolfgang Bangerth Dec 27 '12 at 4:39
    
Do you have any parallel tests that can only run on supercomputers, for example, any large-scale tests against p4est? –  Aron Ahmadia Jan 2 '13 at 22:03

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.