# How to write integration tests for numeric simulation software?

Just to be more precise i'll put a worthy example of my typical use case.

Let's say I'm developing a FEM software that produces several temporal solutions and inserts them in an HDF5 file, along with a bunch of correlated data.

If I use a naive approach to writing integration tests I could do like this:

1. I write the code
2. test it by had on a simple case
3. decide it works and save the right solution
4. I write a custom test that confronts the solution of a new version of the code with the right solution I already found for the simple case.

This obviously won't work, just if I change BLAS implementation every single DOF of my solution could be slightly different. Imposing a test of similarity between solutions within a certain norm could be more effective, but it is not trivial and it feels like overshooting for just writing a test.

How can I write integration tests for this software to be ran automatically on Jenkins?

Is there some tool that helps me deal with the normal uncertainty of the solutions?

• 'Slightly different' should not be an issue as you can always compare solutions/norms and specify some tolerance while doing so. I do this in just a few lines of Bash and it really is trivial. OTOH dealing with code that hangs/doesn't run etc. requires more refined scripts (with timeouts etc.) but is still easily doable in Bash. If you really want you can automate the process using cron that spits out simple html code. – stali Oct 10 '14 at 11:40
• We write these tests to a numeric tolerance that we're comfortable with. You have to make a judgement call about how much of a change is tolerable for your continuous integration testing. – Bill Barth Oct 10 '14 at 15:29
• You can always use numdiff to compare saved and current output within certain tolerances. – Wolfgang Bangerth Oct 11 '14 at 2:43

I don't think you can avoid using a tolerance for floating-point comparisons. Error due to round-off, discretization, and so on using floating-point numbers is unavoidable.

What I typically do to test FEM code I write is:

• test the stiffness & mass matrices on a single element to make sure I get local element assembly right, compare against a known result
• test again on a simple mesh to make sure I get global assembly right, compare against a known result
• test assembling the load vector on a function with known result
• test the whole solver on a manufactured solution that should be "exact" (to working precision, with enough refinement; an example would be a Poisson problem with a linear solution on a $P_{1}$ discretization)
• test the solver on a manufactured solution that will be inexact, refine the solution repeatedly, calculate the error, and regress the error against the mesh size to determine the order of convergence of the solver

For a Poisson problem, these tests exercise most of the solver. For more complicated problems, your mileage may vary; it's easier to write test cases for some problems than it is for others, and it's easier to write tests for some programs (e.g., a serial code you write yourself) than it is for others (e.g., a parallel code, where processor ordering can affect the results of reductions).

All of these tests require some judgment call when it comes to selecting tolerances, but with manufactured solutions, you'll have a good idea of when something is grossly wrong versus when it's correct. Frequently, I start with something like $10^{-7}$ as an absolute tolerance for a solution with characteristic values around 1, and adjust up or down accordingly.

Testing the order of convergence requires the most judgment, because not only do you need to select tolerances, but you need to select the mesh spacings you will use for the regression. It's better to experiment with that first and graph the error versus the mesh size (really, since it's a power law, log of the norm of the error versus log of the mesh size; you will also regress the log of the norm of the error versus the log of the mesh size). Once you have data that makes sense, then convert the code that plots the convergence behavior to a test that compares the order of convergence obtained from regression to the theoretical order of convergence. For example, if I expect an order of convergence of 3, and I get anywhere between, say 2.7 and 3.3, or something to that effect, then I'll say that the test "passed". (Logging the convergence plots somewhere wouldn't hurt either, if you can do that, so if you get a grossly wrong value, you have a sanity check to look at.)

• This is an impeccable answer sir, thank you for the completeness, I'll try to put it in practice on FEM and linear solvers. – Federico Bonelli Oct 10 '14 at 19:44

Geoff has already given an excellent overview, but I wanted to provide another real world look on it. In the deal.II project (http://www.dealii.org/) we run some 7,000 tests with every change in the code base, on multiple platforms.

The tests Geoff describes are mostly "integration tests", i.e., they run through a significant part of the code base. You need relatively few of these, but it turns out that they are very difficult to debug: if they fail, and you don't have the immediate commit that broke them, there is a great deal of code to look through to find out which piece changed functionality. To avoid this, we have a few integration tests, but a very large number (thousands) of small unit tests that simply test one very small part of the library. The typical case where a change breaks functionality is that several of the integration tests fail plus several of the small unit tests, and the simplest of the unit tests is the one one uses for debugging.

As for how these tests work: yes, we store the output on a reference platform and then compare the output on platforms that run the tests. We use numdiff to avoid the issue of small changes in roundoff, with appropriately set tolerances. It took us a very long time of tweaking tolerances to ensure that all 7,000 or so tests run successfully on every platform we use. Before that, we had simply designated one particular machine as the "reference platform", and the tests were saved there and tests run there.

• Numdiff looks like a nice tool – Federico Bonelli Oct 11 '14 at 12:44