I'm a huge advocate of test-driven development in scientific computing. It's utility in practice is just staggering, and really alleviates the classic troubles that code developers know. However, there are inherent difficulties in testing scientific codes that aren't encountered in general programming, so TDD texts aren't terribly useful as tutorials. For example:
In general you don't know an exact answer for a given complex problem a priori, so how can you write a test?
The degree of parallelism changes; I recently encountered a bug where using MPI tasks as a multiple of 3 would fail, but a multiple of 2 worked. Additionally, common testing frameworks don't seem very MPI-friendly due to the very nature of MPI -- you have to re-execute a test binary to alter the number of tasks.
Scientific codes often have a lot of tightly coupled, interdependent and interchangeable parts. We've all seen the legacy code, and we know how tempting it is to forgo good design and use global variables.
Often a numerical method may be an "experiment", or the coder doesn't fully understand how it works and is trying to understand it, so anticipating results is impossible.
Some examples of tests that I write for scientific code:
For time integrators, use a simple ODE with an exact solution, and test that your integrator solves it to within a given accuracy, and the order of accuracy is correct by testing with varying step sizes.
Zero-stability tests: check that a method with 0 boundary/initial conditions remains at 0.
Interpolation tests: given a linear function, assure that an interpolation is correct.
Legacy validation: isolate a chunk of code in a legacy application that is know to be correct, and pull some discrete values out to use for testing.
It still often comes up that I can't figure out how to properly test a given chunk of code, aside from manual trial and error. Can you provide some examples of tests you write for numerical code, and/or general strategies for testing scientific software?