I found that a lot of my computational science programming has testing requirements that are not covered by standard test frameworks:
Computation time testing
- To make sure that algorithms don't get slower. I could do something like
assureSmallerEqual(RuntimeWrapper(algorithm),53)
but I'd like the 53 seconds threshold to be reduced continuously as I am working on the algorithm, i.e. something likeassureSmallerEqual(RuntimeWrapper(algorithm),'previousbest+noisetolerance')
- To make sure that algorithms don't get slower. I could do something like
Performance testing
- To make sure that an algorithm that previously found a good approximation to an analytical solution still finds a solution that is at least as good or better. Again, this could be a emulated by a standard integration test, but I'd like for the tolerance to shrink continuously as the algorithm gets better and better. Think of replacing
assureAlmostEqual(foo(),1,places=3)
byassureAlmostEqual(foo(),1,places='previousbest')
- To make sure that an algorithm that previously found a good approximation to an analytical solution still finds a solution that is at least as good or better. Again, this could be a emulated by a standard integration test, but I'd like for the tolerance to shrink continuously as the algorithm gets better and better. Think of replacing
Physical requirements testing
- To make sure that algorithms don't suddenly need more memory/hard disk space. Very similar to 1.
Abstract requirements testing
- To make sure that an algorithm that did fine with quadratic approximations doesn't suddenly need cubic approximations, or that an algorithm that did fine with time step 0.1 doesn't suddenly need 0.01 for stability. Again, these could be emulated by standard integration tests, but the goal is to remember what the smallest requirement parameter was that achieved a certain goal, so this would require a lot of manual updating. For example, if
foo(10)
previously threw no exceptions, I'd like for the framework to make surefoo(10)
still works and also try iffoo(9)
now works (in which case all future tests would ensurefoo(9)
still works).
- To make sure that an algorithm that did fine with quadratic approximations doesn't suddenly need cubic approximations, or that an algorithm that did fine with time step 0.1 doesn't suddenly need 0.01 for stability. Again, these could be emulated by standard integration tests, but the goal is to remember what the smallest requirement parameter was that achieved a certain goal, so this would require a lot of manual updating. For example, if
One could argue that what I'm asking for doesn't describe tests in the sense of unit/integration testing, since increased runtimes, for example, could be acceptable in return for other improvements.
In practice, however, I know that I would have saved a lot of debugging time if I had the testing functionality above, because in 95% of cases requirements and performance went awry because of bugs I introduced. Indeed, I know for a fact that a lot of bugs that I found (after much time wasted on checking my own code) with external numerical software libraries could have been avoided trivially had the tests above been applied rigorously.
PS
The similarly named question https://stackoverflow.com/questions/34982863/framework-for-regression-testing-of-numerical-code is not a duplicate as it describes functionality that is more easily achievable with standard regression testing frameworks.
The question Strategies for unit testing and test-driven development asks for strategies as opposed to a framework that helps implementing them (and the strategies it asks for/that are provided in the answers are different than what I describe here, in my opinion).