How to write integration tests for numeric simulation software?

Question

Just to be more precise, I'll put a worthy example of my typical use case.

Let's say I'm developing a FEM software that produces several temporal solutions and inserts them in an HDF5 file, along with a bunch of correlated data.

If I use a naive approach to writing integration tests, I could do like this:

1. Write the code
2. Test it by hand on a simple case
3. Decide it works and save the right solution
4. Write a custom test that compares the solution of a new version of the code with the right solution I already found for the simple case.

This obviously won't work, as even changing my BLAS implementation could slightly change every single DOF of my solution. Imposing a test of similarity between solutions within a certain norm could be more effective, but it is not trivial and it feels like overshooting for just writing a test.

How can I write integration tests for this software to be run automatically on Jenkins? Is there some tool that helps me deal with the normal uncertainty of the solutions?

Answer 1

I don't think you can avoid using a tolerance for floating-point comparisons. Error due to round-off, discretization, and so on using floating-point numbers is unavoidable.

What I typically do to test FEM code I write is:

• test the stiffness & mass matrices on a single element to make sure I get local element assembly right, compare against a known result
• test again on a simple mesh to make sure I get global assembly right, compare against a known result
• test assembling the load vector on a function with known result
• test the whole solver on a manufactured solution that should be "exact" (to working precision, with enough refinement; an example would be a Poisson problem with a linear solution on a $P_{1}$ discretization)
• test the solver on a manufactured solution that will be inexact, refine the solution repeatedly, calculate the error, and regress the error against the mesh size to determine the order of convergence of the solver

For a Poisson problem, these tests exercise most of the solver. For more complicated problems, your mileage may vary; it's easier to write test cases for some problems than it is for others, and it's easier to write tests for some programs (e.g., a serial code you write yourself) than it is for others (e.g., a parallel code, where processor ordering can affect the results of reductions).

All of these tests require some judgment call when it comes to selecting tolerances, but with manufactured solutions, you'll have a good idea of when something is grossly wrong versus when it's correct. Frequently, I start with something like $10^{-7}$ as an absolute tolerance for a solution with characteristic values around 1, and adjust up or down accordingly.

Testing the order of convergence requires the most judgment, because not only do you need to select tolerances, but you need to select the mesh spacings you will use for the regression. It's better to experiment with that first and graph the error versus the mesh size (really, since it's a power law, log of the norm of the error versus log of the mesh size; you will also regress the log of the norm of the error versus the log of the mesh size). Once you have data that makes sense, then convert the code that plots the convergence behavior to a test that compares the order of convergence obtained from regression to the theoretical order of convergence. For example, if I expect an order of convergence of 3, and I get anywhere between, say 2.7 and 3.3, or something to that effect, then I'll say that the test "passed". (Logging the convergence plots somewhere wouldn't hurt either, if you can do that, so if you get a grossly wrong value, you have a sanity check to look at.)

Answer 2

Geoff has already given an excellent overview, but I wanted to provide another real world look on it. In the deal.II project (http://www.dealii.org/) we run some 7,000 tests with every change in the code base, on multiple platforms.

The tests Geoff describes are mostly "integration tests", i.e., they run through a significant part of the code base. You need relatively few of these, but it turns out that they are very difficult to debug: if they fail, and you don't have the immediate commit that broke them, there is a great deal of code to look through to find out which piece changed functionality. To avoid this, we have a few integration tests, but a very large number (thousands) of small unit tests that simply test one very small part of the library. The typical case where a change breaks functionality is that several of the integration tests fail plus several of the small unit tests, and the simplest of the unit tests is the one one uses for debugging.

As for how these tests work: yes, we store the output on a reference platform and then compare the output on platforms that run the tests. We use numdiff to avoid the issue of small changes in roundoff, with appropriately set tolerances. It took us a very long time of tweaking tolerances to ensure that all 7,000 or so tests run successfully on every platform we use. Before that, we had simply designated one particular machine as the "reference platform", and the tests were saved there and tests run there.

Answer 3

A few additional points I would like to add to other answers.

1. Corner test cases should be part of the regression test suite - ill conditioned problems, ill conditioned - bad aspect ratios, orthotropic and un-isotropic material properties, improperly constrained models.
2. Make sure reaction forces match.
3. Sturm checks for eigenvalue problems especially when there are multiple eigenvalues.

4. In case of iterative solvers, one might get the correct answer but numerical bugs, or incorrect pre-conditioner construction/factoringmight make the solver slow to converge. So performance needs to be compared with the old code for the same stopping tolerance setting.

5. Stopping tolerance needs to be tightened progressively and number of iterations noted.

6. Paralle and GPU: Performance scaling for number of threads, cache sizes, etc. need to be cross-checked and should be as expected.

PS:

7. Write a short and simple matrix-vector multiply and check the norm ||Ax*-b|| where x* is the answer returned by the solver. This has to be simple and short so that its correctness can be verified in an hour or two by doing a code walk-through with the help of a couple of co-workers.