Is it worthwhile to write unit tests for scientific research codes?

Question

I am strongly convinced of the value of using tests that verify a complete program (e.g. convergence tests), including an automated set of regression tests. After reading some programming books, I've gotten the nagging feeling that I "ought to" write unit tests (i.e., tests that verify the correctness of a single function and do not amount to running the whole code to solve a problem) as well. However, unit tests don't always seem to fit with scientific codes, and end up feeling artificial or like a waste of time.

Should we write unit tests for research codes?

Answer 1

For many years I was under the misapprehension that I didn't have enough time to write unit tests for my code. When I did write tests, they were bloated, heavy things which only encouraged me to think that I should only ever write unit tests when I knew they were needed.

Then I started to use Test Driven Development and I found it to be a complete revelation. I'm now firmly convinced that I don't have the time not to write unit-tests.

In my experience, by developing with testing in mind you end up with cleaner interfaces, more focussed classes & modules and generally more SOLID, testable code.

Every time I work with legacy code which doesn't have unit tests and have to manually test something, I keep thinking "this would be so much quicker if this code already had unit tests". Every time I have to try and add unit test functionality to code with high coupling, I keep thinking "this would be so much easier if it had been written in a de-coupled way".

Comparing and contrasting the two experimental stations that I support. One has been around for a while and has a great deal of legacy code, while the other is relatively new.

When adding functionality to the old lab, it is often a case of getting down to the lab and spending many hours working through the implications of the functionality they need and how I can add that functionality without affecting any of the other functionality. The code is simply not set up to allow off-line testing, so pretty much everything has to be developed on-line. If I did try to develop off-line then I would end up with more mock objects than would be reasonable.

In the newer lab, I can usually add functionality by developing it off-line at my desk, mocking out only those things which are immediately required, and then only spending a short time in the lab, ironing out any remaining problems not picked up off-line.

For clarity, and since @naught101 asked...

I tend to work on experimental control and data acquisition software, with some ad hoc data analysis, so the combination of TDD with revision control helps to document both changes in the underlying experiment hardware and as well as changes in data collection requirements over time.

Even in the situation of developing exploratory code however, I could see a significant benefit from having assumptions codified, along with the ability to see how those assumptions evolve over time.

Answer 2

Scientific codes tend to have constellations of interlocking functions more often than the business codes I have worked on, usually due to the mathematical structure of the problem. So, I do not think unit tests for individual functions are very effective. However, I do think there is a class of unit tests which are effective, and are still quite different from whole program tests in that they target specific functionality.

I just briefly define what I mean by these kinds of tests. Regression testing looks for changes in existing behavior (validated somehow) when changes are made to the code. Unit testing runs a piece of code and checks that it gives a desired output based upon a specification. They are not that different, as the original regression test was a unit test since I had to determine that the output was valid.

My favorite example of a numerical unit test is testing the convergence rate of a finite element implementation. It is definitely not simple, but it takes a known solution to a PDE, runs several problems at decreasing mesh size $h$, and then fits the error norm to the curve $C h^r$ where $r$ is the convergence rate. I do this for the Poisson problem in PETSc using Python. I am not looking for a difference, as in regression, but a particularly rate $r$ specified for the given element.

Two more examples of unit testing, coming from PyLith, are point location, which is a single function which is easy to produce synthetic results for, and creation of zero volume cohesive cells in a mesh, which involves several functions but addresses a circumscribed piece of functionality in the code.

There are many tests of this kind, including conservation and consistency tests. The operation is not that different from regression (you run a test and check the output against a standard), but the standard output comes from a specification rather than a previous run.

Answer 3

Ever since I read about Test-Driven Development in Code Complete, 2nd edition, I've used a unit testing framework as part of my development strategy, and it's dramatically increased my productivity by reducing the amount of time I spent debugging because the various tests I write are diagnostic. As a side benefit, I'm much more confident in my scientific results, and have used my unit tests on a number of occasions to defend my results. If there is an error in a unit test, I can usually figure out why pretty quickly. If my application crashes and all of my unit tests pass, I do a code coverage analysis to see what parts of my code aren't exercised, as well as step through the code with a debugger to pinpoint the source of the error. Then I write a new test to make sure that the bug stays fixed.

Many of the tests I write aren't pure unit tests. Strictly defined, unit tests are supposed to exercise the functionality of one function. When I can test easily a single function using mock data, I do that. Other times, I can't easily mock the data I need to write a test that exercises the functionality of a given function, so I'll test that function along with others in an integration test. Integration tests test the behavior of multiple functions at once. As Matt points out, scientific codes are often a constellation of interlocking functions, but often times, certain functions are called in sequence, and unit tests can be written to test the output at intermediate steps. For example, if my production code calls five functions in sequence, I'll write five tests. The first test will call the first function only (so it's a unit test). Then second test will call the first and second functions, the third test will call the first three functions, and so on. Even if I could write unit tests for every single function in my code, I would write integration tests anyway, because bugs can arise when various modular pieces of a program are combined. Finally, after writing all of the unit tests and integration tests I think I need, I'll wrap my case studies in unit tests and use them for regression testing, because I want my results to be repeatable. If they're not repeatable, and I get different results, I want to know why. The failure of a regression test may not be a real issue, but it will force me to figure out if the new results are at least as trustworthy as the old results.

Also worthwhile along with unit testing are static code analysis, memory debuggers, and compiling with compiler warning flags to catch simple errors and unused code.

Answer 4

In my experience, as the complexity of scientific research codes increases, there is a need to have a very modular approach in programming. This can be painful for codes with a large and ancient based (f77 anyone?) but it is necessary moving forward. As a module gets built around a specific aspect of the code (for CFD applications, think Boundary Conditions or Thermodynamics), unit testing is very valuable to validate the new implementation and isolate issues and further software developments.

These unit tests should be one level below code verification (can I recover the analytical solution of my wave equation?) and 2 levels below code validation (can I predict the correct peak RMS values in my turbulent pipe flow), simply insuring that the programming (are the arguments correctly passed, are the pointers pointing to the right thing?) and the "math" (this subroutine computes the friction coefficient. If I input a set of numbers and compute the solution by hand, does the routine yield the same result?) are correct. Basically going one level above what the compilers can spot, i.e. basic syntax errors.

I would definitely recommend it for at least some crucial modules in your application. However, one has to realize that it is extremely tedious and time-consuming so unless you have unlimited man-power, I wouldn't recommend it for 100% of a complex code.

Answer 5

Unit testing for scientific codes is useful for a variety of reasons.

Three in particular are:

• Unit tests help other people understand the constraints of your code. Basically, unit tests are a form of documentation.

• Unit tests check to make sure that a single unit of code is returning correct results, and check to make sure that the behavior of a program doesn’t change when the details are modified.

• Using unit tests makes it easier to modularise your research codes. This can be particularly important if you start trying to target your code at a new platform, for instance you are interested in parallelising it, or running it on a GPGPU machine.

Most of all, unit tests give you confidence that the research results you are producing using your codes are valid and verifiable.

I note that you mention regression testing in your question. In many cases, regression testing is achieved through the automated, regular execution of unit tests and/or integration tests (which test that pieces of code work correctly when combined; in scientific computing, this is often done by comparing output to experimental data or the results of earlier programs that are trusted). It sounds like you are already using integration tests or unit test at the level of large complex components successfully.

What I would say is that as research codes get increasingly complex, and rely on other people's code and libraries, it is important to understand where the error occurs when it does. Unit testing allows the error to pinpointed much more easily.

You may find the description, evidence and references in Section 7 "Plan for mistakes" of the paper I co-authored on Best Practices for Scientific Computing useful - it also introduces the complementary concept of defensive programming.

Answer 6

In my deal.II classes I teach that software that does not have tests does not work correctly (and go on to stress that I purposefully said "does not work correctly", not "may not work correctly).

Of course I live by the mantra -- which is how deal.II has come to run 2,500 tests with every commit ;-) [Update as of 2022: It is now 13,500 tests for each commit.]

More seriously, I think Matt already defines the two classes of tests well. We write unit tests for the lower level stuff and it sort of naturally progresses to regression tests for the higher level stuff. I don't think I could draw a clear boundary that would separate our tests to one side or the other, there are certainly many that tread the line where someone has looked at the output and found it to be largely reasonable (unit test?) without having looked at it to the last bit of accuracy (regression test?).

Answer 7

Yes and no. Certainly unittest for fundamental routines of the basic toolset you use to make your life easier, such as conversion routines, string mappings, basic physics and math, etc. When it comes to calculations classes or functions, they may generally require long run times, and you may actually prefer to test them as functional tests, instead of as units. Also, unittest and stress a lot those classes and entities whose level and use are going to change a lot (e.g. for optimization purposes) or whose internal details are going to be changed for whatever reason. The most typical example is a class wrapping a huge matrix, mapped from disk.

Answer 8

Absolutely!

What, that's not enough for you?

In scientific programming more than any other kind, we're developing based on trying to match a physical system. How will you know if you've done that other than by testing? Before you even start coding, decide how you're going to use your code and work out a few example runs. Try to catch any possible edge cases. Do it in a modular way - for example, for a neural network you might make a set of tests for a single neuron and a set of tests for a complete neural network. That way when you start writing code you can make sure your neuron works before you start working on the network. Working in stages like this means that when you run into a problem you only have the most recent 'stage' of code to test, the earlier stages have already been tested.

Plus, once you have the tests, if you need to rewrite the code in a different language (converting to CUDA, for example), or even if you're just updating it, you already have the testcases and you can use them to make sure that both versions of your program work the same way.

Answer 9

I'd approach this question pragmatically rather than dogmatically. Ask yourself the question: "What could go wrong in function X?" Imagine what happens to the output when you introduce some typical bugs into the code: a wrong prefactor, a wrong index, ... And then write unit tests that are likely to detect that kind of bug. If for a given function there is no way to write such tests without repeating the code of the function itself, then don't - but think about tests on the next higher level.

A much more important problem with unit tests (or in fact any tests) in scientific code is how to deal with the uncertainties of floating-point arithmetic. As far as I know there are no good general solutions yet.

Answer 10

The mantra is "Untested code is broken code" and that came from the boss. Rather than repeat all the good reasons for doing unit testing, I want to just add a few specifics.

• Memory usage should be tested. Every function which allocates memory should be tested making sure that the functions that store and retrieve data into that memory are doing the right thing. This is even more important in the GPU world.
• While briefly mentioned before, testing edge cases is extremely important. Think of these tests the same way you test the result of any calculation. Make sure the code behaves at the edges and fails gracefully (however you define that in your simulation) when input parameters or data falls outside acceptable boundaries. The thinking involved in writing this sort of test helps sharpen your work and may be one of the reasons that you rarely find someone who has written unit tests but does not find the process useful.
• Use a test framework (as mentioned by Geoff who provided a nice link). I have used the BOOST test framework in conjunction with the CTest system of CMake and can recommend it as an easy way to quickly write unit tests (as well as validation and regression tests).

Answer 11

Yes.

The idea that any code is written without unit tests is anathema. Unless you prove your code correct and then prove the proof correct =P.

Answer 12

I've used unit testing to good effect on several small scale (i.e. single programmer) codes, including the third version of my dissertation analysis code in particle physics.

The first two versions had collapsed under their own weight and the multiplication of interconnections.

Other have written that the interaction between modules are often the place where scientific coding breaks, and they're right about that. But it is a lot easier to diagnose those problems when you can show conclusively that each module is doing what it is meant to do.

Answer 13

A slightly different approach that I used whilst developing a chemical solver (for complex geological domains) was what you could call Unit Testing by Copy and Paste Snippet.

Building a test harness for the original code embedded in a large chemical system modeller was not feasible in the timeframe.

However, I was able to work up an increasingly complex set of snippets showing how the (Boost Spirit) parser for the chemical formulas worked, as unit tests for different expressions.

The final, most complex unit test was very close to the code needed in the system, without having to change that code to be mockable. I was thus able to copy across my unit tested code.

What makes this more than just a learning exercise and a true regression suite are two factors - the unit tests kept in the main source and run as part of other tests for that application (and yes, they did pick up a side-effect from Boost Spirit changing 2 years later) - because the code copied and pasted across was minimally modified in the real application, it could have comments referring back to the unit tests to help someone keep them in synch.

Answer 14

For bigger code bases, tests (not necessarily unit tests) for the highlevel stuff are useful. Unit tests for some simpler algorithms are useful as well to make sure your code isn't doing nonsense because your helper function is using sin instead of cos.

But for the overall research code it is very hard to write and maintain tests. Algorithms tend to be large without meaningful intermediate results which can have obvious tests and often take a long time to run before there is a result. Of course you can test against reference runs which had nice results, but this is not a good test in the unit test sense.

Results often are approximations of the true solution. While you can test your simple functions if they are accurate up to some epsilon, it will be very hard to verify if i.e. some result mesh is correct or not, which was evaluated by visual inspection by the user (you) before.

In such cases automated tests often have a too high cost/benefit ratio. I recommend something better: Write test programs. For example I wrote a medium size python script to create data about results, like histograms of edge sizes and angles of a mesh, area of the biggest and smallest triangle and their ratio, etc.

I can both use it to evaluate input and output meshes during normal operation and use it to have a sanity check after I changed the algorithm. When I change the algorithm I do not always know if the new result is better, because there often is no absolute measure which approximation is the best. But by generating such metrics I can tell about some factors what is better like "The new variant eventually has a better angle ratio but a worse convergence rate".