# Strategies for unit testing and test-driven development

I'm a huge advocate of test-driven development in scientific computing. It's utility in practice is just staggering, and really alleviates the classic troubles that code developers know. However, there are inherent difficulties in testing scientific codes that aren't encountered in general programming, so TDD texts aren't terribly useful as tutorials. For example:

• In general you don't know an exact answer for a given complex problem a priori, so how can you write a test?

• The degree of parallelism changes; I recently encountered a bug where using MPI tasks as a multiple of 3 would fail, but a multiple of 2 worked. Additionally, common testing frameworks don't seem very MPI-friendly due to the very nature of MPI -- you have to re-execute a test binary to alter the number of tasks.

• Scientific codes often have a lot of tightly coupled, interdependent and interchangeable parts. We've all seen the legacy code, and we know how tempting it is to forgo good design and use global variables.

• Often a numerical method may be an "experiment", or the coder doesn't fully understand how it works and is trying to understand it, so anticipating results is impossible.

Some examples of tests that I write for scientific code:

• For time integrators, use a simple ODE with an exact solution, and test that your integrator solves it to within a given accuracy, and the order of accuracy is correct by testing with varying step sizes.

• Zero-stability tests: check that a method with 0 boundary/initial conditions remains at 0.

• Interpolation tests: given a linear function, assure that an interpolation is correct.

• Legacy validation: isolate a chunk of code in a legacy application that is know to be correct, and pull some discrete values out to use for testing.

It still often comes up that I can't figure out how to properly test a given chunk of code, aside from manual trial and error. Can you provide some examples of tests you write for numerical code, and/or general strategies for testing scientific software?

• Could you, please, clarify what you mean by interpolation tests? – Dmitry Kabanov Jun 23 '15 at 9:16

Verify through refinement studies that the method achieves the theoretical order of accuracy.

Conservation of answer. Bit-wise and norm-wise reproduction of solutions.

• I meant to mention MMS in the original post; it's good for code verification, but from a unit testing perspective it's entirely worthless. If those tests fail, it provides no clue as to where or why. – Aurelius Sep 13 '13 at 12:28
• @Aurelius: But it's a great strategy for test-driven development! For PDE/ODE/Linear algebra codes, you should write very small MMS tests that can run in less than a second. When you make a change, you run them. If they break, you did something wrong! You'd be surprised how much a $2\times 2\times 2$-element problem can tell you (or whatever). – Bill Barth Sep 13 '13 at 13:15
• A lot of the literature I've seen on MMS are basically global solutions, e.g. for CFD problems, a manufactured solution might be an airfoil analysis. When this test fails, at best you've narrowed down the culprit to 5,000 lines of code, so it's pretty worthless for TDD - you have no clue where the actual failure comes about. I agree a 2x2x2 problem is extremely valuable, and I personally use them a lot. But it's pretty common that I encounter problems that only pop up with larger systems; I actually found an ifort compiler bug recently that only manifested in large problems. – Aurelius Sep 13 '13 at 17:29
• @Aurelius: No argument here. You should have a range of tests and run them all frequently. – Bill Barth Sep 13 '13 at 19:02
• @Aurelius At face-value, MMS is not a unit test, but a functional or acceptance test (ie of the whole system). However, codes often have separate stages (or can be divided into them). eg advection, pressure, viscosity. One could then test only one of these stages (a "unit"). Similarly, a code could be tested without a BC, and then with one. A friend did his PhD on unit testing, and he reckoned the greatest benefit was it forced you to break up your program into units, so it can be unit-tested... perhaps this is more applicable here than it seems at first (and in other ways I don't know of). – hyperpallium Nov 20 '17 at 7:53