Are there testing frameworks for numerical software development

Question

I found that a lot of my computational science programming has testing requirements that are not covered by standard test frameworks:

1. Computation time testing

   • To make sure that algorithms don't get slower. I could do something like assureSmallerEqual(RuntimeWrapper(algorithm),53) but I'd like the 53 seconds threshold to be reduced continuously as I am working on the algorithm, i.e. something like assureSmallerEqual(RuntimeWrapper(algorithm),'previousbest+noisetolerance')

2. Performance testing

   • To make sure that an algorithm that previously found a good approximation to an analytical solution still finds a solution that is at least as good or better. Again, this could be a emulated by a standard integration test, but I'd like for the tolerance to shrink continuously as the algorithm gets better and better. Think of replacing assureAlmostEqual(foo(),1,places=3) by assureAlmostEqual(foo(),1,places='previousbest')

3. Physical requirements testing

   • To make sure that algorithms don't suddenly need more memory/hard disk space. Very similar to 1.

4. Abstract requirements testing

   • To make sure that an algorithm that did fine with quadratic approximations doesn't suddenly need cubic approximations, or that an algorithm that did fine with time step 0.1 doesn't suddenly need 0.01 for stability. Again, these could be emulated by standard integration tests, but the goal is to remember what the smallest requirement parameter was that achieved a certain goal, so this would require a lot of manual updating. For example, if foo(10) previously threw no exceptions, I'd like for the framework to make sure foo(10) still works and also try if foo(9) now works (in which case all future tests would ensure foo(9) still works).

One could argue that what I'm asking for doesn't describe tests in the sense of unit/integration testing, since increased runtimes, for example, could be acceptable in return for other improvements.
In practice, however, I know that I would have saved a lot of debugging time if I had the testing functionality above, because in 95% of cases requirements and performance went awry because of bugs I introduced. Indeed, I know for a fact that a lot of bugs that I found (after much time wasted on checking my own code) with external numerical software libraries could have been avoided trivially had the tests above been applied rigorously.

PS

The similarly named question https://stackoverflow.com/questions/34982863/framework-for-regression-testing-of-numerical-code is not a duplicate as it describes functionality that is more easily achievable with standard regression testing frameworks.

The question Strategies for unit testing and test-driven development asks for strategies as opposed to a framework that helps implementing them (and the strategies it asks for/that are provided in the answers are different than what I describe here, in my opinion).

Answer 1

1. This type of test appears to me to be poorly defined because its test condition is tied to the particular machine on which you did tests in development. One of the points of testing is that running your tests on my laptop tells me whether there's something wrong with the code or the environment I've set up. The 53 seconds is specific to your development machine, and the running time will also increase if the testing machine is under load from other workloads or users. I would not expect testing frameworks to address this: "function runs on input in under 53 seconds" is just not a very good correctness spec.

2. I think this ambiguous and undesirable from the point of view of software testing for the same reasons 1, you lose the pass-or-fail justification for software testing.

3. This is quite common, let me describe one solution. It's not quite the job of a testing framework, but you can use a separate tool as described in the Unix SE question Limit memory usage for a single Linux process. One standard tool to try first is the ulimit command in bash, which lets you run a process and make sure it crashes if it tries to, e.g., allocate too much memory. So if you run the runtests script with a memory limit, it will crash and the testing framework should be able to handle that as a regular test failure.

4. Most testing frameworks do not think of unit tests this way at all. The test suite is run (e.g., before committing code to master or before deploying), and the result is a yes or a no indicating whether it functions. Testing frameworks don't consider it part of their job to, e.g., track feature progress, and that's not what testing generally is. What you'd do here is you'd write two tests expect_succeeds(foo(10)); expect_fails(foo(9)). Each time, both tests are run, and the successes and expected failures pass. When you implement foo(9) and it succeeds, the expects-failure test now fails, so you'd rewrite expect_succeeds(foo(9)), and this is an absolutely standard feature of all frameworks. But you must be explicit about what behaviour you expect, because otherwise it just goes against too much against the basic ideas of software testing.

There is an alternative approach to all this. You are trying to get the testing framework to do extra work in tracking the iterative progress of your code, but testing frameworks work, and are expected to work, on snapshots of code, giving pass-or-fail answers. It might be easier to take an algorithm $A$, create a test suite for $A$ as it is, then make a full copy of $A$ called $B$, have tests performs_better(foo_A(), foo_B()), and just keep working on $B$. Now (a) the testing framework will then have no trouble comparing $A$ and $B$, and (b) there is no longer any sense of comparing code to how it used to be, all code and tests are now immutable and unambiguous. This is similar in spirit to how one might handle system rewrites.