# Are there testing frameworks for numerical software development

I found that a lot of my computational science programming has testing requirements that are not covered by standard test frameworks:

1. Computation time testing

• To make sure that algorithms don't get slower. I could do something like assureSmallerEqual(RuntimeWrapper(algorithm),53) but I'd like the 53 seconds threshold to be reduced continuously as I am working on the algorithm, i.e. something like assureSmallerEqual(RuntimeWrapper(algorithm),'previousbest+noisetolerance')
2. Performance testing

• To make sure that an algorithm that previously found a good approximation to an analytical solution still finds a solution that is at least as good or better. Again, this could be a emulated by a standard integration test, but I'd like for the tolerance to shrink continuously as the algorithm gets better and better. Think of replacing assureAlmostEqual(foo(),1,places=3) by assureAlmostEqual(foo(),1,places='previousbest')
3. Physical requirements testing

• To make sure that algorithms don't suddenly need more memory/hard disk space. Very similar to 1.
4. Abstract requirements testing

• To make sure that an algorithm that did fine with quadratic approximations doesn't suddenly need cubic approximations, or that an algorithm that did fine with time step 0.1 doesn't suddenly need 0.01 for stability. Again, these could be emulated by standard integration tests, but the goal is to remember what the smallest requirement parameter was that achieved a certain goal, so this would require a lot of manual updating. For example, if foo(10) previously threw no exceptions, I'd like for the framework to make sure foo(10) still works and also try if foo(9) now works (in which case all future tests would ensure foo(9) still works).

One could argue that what I'm asking for doesn't describe tests in the sense of unit/integration testing, since increased runtimes, for example, could be acceptable in return for other improvements.
In practice, however, I know that I would have saved a lot of debugging time if I had the testing functionality above, because in 95% of cases requirements and performance went awry because of bugs I introduced. Indeed, I know for a fact that a lot of bugs that I found (after much time wasted on checking my own code) with external numerical software libraries could have been avoided trivially had the tests above been applied rigorously.

PS

The similarly named question https://stackoverflow.com/questions/34982863/framework-for-regression-testing-of-numerical-code is not a duplicate as it describes functionality that is more easily achievable with standard regression testing frameworks.

The question Strategies for unit testing and test-driven development asks for strategies as opposed to a framework that helps implementing them (and the strategies it asks for/that are provided in the answers are different than what I describe here, in my opinion).

• Is the numerical software for simulation or for analysis of experimental data? – mathew gunther Aug 6 '19 at 21:12
• @mathewgunther Numerical Analysis/ Numerical Algebra. No data analysis – Bananach Aug 6 '19 at 22:53
• I know that a lot of big simulation companies use frameworks that they created on their own. Basically in python. You need to have test cases which are started by the python scripts and write out some results. Afterwards the results can be compared with some kind of reference and output a report. The test can be automized an run daily or weekly or monthly etc. Not sure if there is some kind of generel framework as ever simulation software is kind of special in implementation etc. – vydesaster Aug 12 '19 at 21:11

3. This is quite common, let me describe one solution. It's not quite the job of a testing framework, but you can use a separate tool as described in the Unix SE question Limit memory usage for a single Linux process. One standard tool to try first is the ulimit command in bash, which lets you run a process and make sure it crashes if it tries to, e.g., allocate too much memory. So if you run the runtests script with a memory limit, it will crash and the testing framework should be able to handle that as a regular test failure.
4. Most testing frameworks do not think of unit tests this way at all. The test suite is run (e.g., before committing code to master or before deploying), and the result is a yes or a no indicating whether it functions. Testing frameworks don't consider it part of their job to, e.g., track feature progress, and that's not what testing generally is. What you'd do here is you'd write two tests expect_succeeds(foo(10)); expect_fails(foo(9)). Each time, both tests are run, and the successes and expected failures pass. When you implement foo(9) and it succeeds, the expects-failure test now fails, so you'd rewrite expect_succeeds(foo(9)), and this is an absolutely standard feature of all frameworks. But you must be explicit about what behaviour you expect, because otherwise it just goes against too much against the basic ideas of software testing.
There is an alternative approach to all this. You are trying to get the testing framework to do extra work in tracking the iterative progress of your code, but testing frameworks work, and are expected to work, on snapshots of code, giving pass-or-fail answers. It might be easier to take an algorithm $$A$$, create a test suite for $$A$$ as it is, then make a full copy of $$A$$ called $$B$$, have tests performs_better(foo_A(), foo_B()), and just keep working on $$B$$. Now (a) the testing framework will then have no trouble comparing $$A$$ and $$B$$, and (b) there is no longer any sense of comparing code to how it used to be, all code and tests are now immutable and unambiguous. This is similar in spirit to how one might handle system rewrites.