20
$\begingroup$

I'm a huge advocate of test-driven development in scientific computing. It's utility in practice is just staggering, and really alleviates the classic troubles that code developers know. However, there are inherent difficulties in testing scientific codes that aren't encountered in general programming, so TDD texts aren't terribly useful as tutorials. For example:

  • In general you don't know an exact answer for a given complex problem a priori, so how can you write a test?

  • The degree of parallelism changes; I recently encountered a bug where using MPI tasks as a multiple of 3 would fail, but a multiple of 2 worked. Additionally, common testing frameworks don't seem very MPI-friendly due to the very nature of MPI -- you have to re-execute a test binary to alter the number of tasks.

  • Scientific codes often have a lot of tightly coupled, interdependent and interchangeable parts. We've all seen the legacy code, and we know how tempting it is to forgo good design and use global variables.

  • Often a numerical method may be an "experiment", or the coder doesn't fully understand how it works and is trying to understand it, so anticipating results is impossible.

Some examples of tests that I write for scientific code:

  • For time integrators, use a simple ODE with an exact solution, and test that your integrator solves it to within a given accuracy, and the order of accuracy is correct by testing with varying step sizes.

  • Zero-stability tests: check that a method with 0 boundary/initial conditions remains at 0.

  • Interpolation tests: given a linear function, assure that an interpolation is correct.

  • Legacy validation: isolate a chunk of code in a legacy application that is know to be correct, and pull some discrete values out to use for testing.

It still often comes up that I can't figure out how to properly test a given chunk of code, aside from manual trial and error. Can you provide some examples of tests you write for numerical code, and/or general strategies for testing scientific software?

Improve this question
$\endgroup$
1
  • $\begingroup$ Could you, please, clarify what you mean by interpolation tests? $\endgroup$
    – Dmitry Kabanov
    Commented Jun 23, 2015 at 9:16

3 Answers 3

Reset to default
10
$\begingroup$

Method of manufactured solutions.

Verify through refinement studies that the method achieves the theoretical order of accuracy.

Conservation of answer. Bit-wise and norm-wise reproduction of solutions.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I meant to mention MMS in the original post; it's good for code verification, but from a unit testing perspective it's entirely worthless. If those tests fail, it provides no clue as to where or why. $\endgroup$
    – Aurelius
    Commented Sep 13, 2013 at 12:28
  • 6
    $\begingroup$ @Aurelius: But it's a great strategy for test-driven development! For PDE/ODE/Linear algebra codes, you should write very small MMS tests that can run in less than a second. When you make a change, you run them. If they break, you did something wrong! You'd be surprised how much a $2\times 2\times 2$-element problem can tell you (or whatever). $\endgroup$
    – Bill Barth
    Commented Sep 13, 2013 at 13:15
  • $\begingroup$ A lot of the literature I've seen on MMS are basically global solutions, e.g. for CFD problems, a manufactured solution might be an airfoil analysis. When this test fails, at best you've narrowed down the culprit to 5,000 lines of code, so it's pretty worthless for TDD - you have no clue where the actual failure comes about. I agree a 2x2x2 problem is extremely valuable, and I personally use them a lot. But it's pretty common that I encounter problems that only pop up with larger systems; I actually found an ifort compiler bug recently that only manifested in large problems. $\endgroup$
    – Aurelius
    Commented Sep 13, 2013 at 17:29
  • $\begingroup$ @Aurelius: No argument here. You should have a range of tests and run them all frequently. $\endgroup$
    – Bill Barth
    Commented Sep 13, 2013 at 19:02
  • $\begingroup$ @Aurelius At face-value, MMS is not a unit test, but a functional or acceptance test (ie of the whole system). However, codes often have separate stages (or can be divided into them). eg advection, pressure, viscosity. One could then test only one of these stages (a "unit"). Similarly, a code could be tested without a BC, and then with one. A friend did his PhD on unit testing, and he reckoned the greatest benefit was it forced you to break up your program into units, so it can be unit-tested... perhaps this is more applicable here than it seems at first (and in other ways I don't know of). $\endgroup$
    – hyperpallium
    Commented Nov 20, 2017 at 7:53
7
$\begingroup$

Bill has already listed a few methods which address your concerns.

Addressing your third point, no, there is no reason to introduce strong coupling between parts. Just the opposite: if your functions or classes have well defined interfaces, it will be much easier to exchange for instance a linear solver for another, or a time stepping scheme. Just resist it, and then you will be able to test these components separately. We have done this with deal.II for decades.

To your fourth point: if your method is an experiment, your experiments with the method constitute a test. As long as you do not have analysis, you will have to take these test results as best available. But usually, you have an expectation for instance for the order of a method, or you would know it is exact for a certain class of solutions, for instance polynomials up to a certain degree. Verifying these should be part of your experiments, and as analysis improves, tests can be added.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ To add to Guido's answer, the experience he speaks from is encoded in the ~3,000 tests we run on deal.II after every change: dealii.org/developer/development/… . On the question of what to do if you don't know the exact answer: write a test anyway and let it compare the answer today to the answer yesterday (or whenever you wrote the test). Having a way to spot changes in a code's output is valuable even if you don't know whether they made the answer incorrect or corrected a previously incorrect answer. $\endgroup$
    – Wolfgang Bangerth
    Commented Sep 15, 2013 at 17:22
3
$\begingroup$

I recently found this thesis on TDD in Computational Science. I haven't read it yet so I have no idea if it is any good, but hopefully it can be of some help.

http://cyber.ua.edu/files/2014/12/u0015_0000001_0001551.pdf

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I skimmed some of the intro and conclusions, and assuming a quality level on par with standard PhD thesis, this both explains the process ( in a high level way) and gives actual measurements as to its effectiveness. I think this is quite a find. $\endgroup$
    – Godric Seer
    Commented Jun 1, 2015 at 20:16
  • 1
    $\begingroup$ The link is dead. Did you mean: Nanthaamornphong, A. “The effectiveness of test-driven development and refactoring techniques in computational science and engineering software development”. PhD diss., Uni. Alabama (2014). $\endgroup$
    – AlQuemist
    Commented Aug 14, 2019 at 11:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.