Usually, I write serial code, and when I do, I write unit tests with some xUnit-style testing framework (MATLAB xUnit, PyUnit/nose, or Google's C++ testing framework).
Based on a cursory Google search, I haven't seen much on how practitioners unit test code that uses MPI. Are there any best practices for that?
Compared to Strategies for unit testing and test-driven development, I'm looking for answers relating to what software I should use for a testing framework (if any exist -- the answer could very well be "roll your own code", in which case examples of custom testing code would be helpful).
Most of what I'm looking to test are right-hand side function evaluations and Jacobian matrix assembly routines for time steppers that will be integrating semi-discretized PDEs. I will be using PETSc, so if there's anything PETSc-specific, that would be helpful in addition to more general testing frameworks.
Clarification Edits:
An example would be in ${PETSC_DIR}/src/ts/examples/tutorials/ex2.c
, where I would want to test something like RHSFunction
(a right-hand side function evaluation) and RHSJacobian
(a Jacobian matrix evaluation). I'd be testing against known values for the assembled right-hand side and assembled Jacobian matrix; I can obtain these values analytically for some simple problem instances. These functions are application-specific functions that won't exercise any other application-level function, but they could call MPI if vector or matrix assembly is done within the function (as in the linked PETSc example above). If I write functions that only calculate portions of vectors or matrices local to a processor, I would want to test against the global, assembled version if possible because, being new to parallel programming, it's more intuitive for me to think of global vectors and global matrices. These tests would be run on small problem sizes and small numbers of processors.
I can think of a few strategies to do this:
- A strategy that probably won't work well, based on the Google searches I've done on this topic, would be to construct a known output, find the relative/absolute error in parallel, and then do naive comparisons. The output will probably be garbled -- anyone who has written a "Hello, world" program with MPI knows why -- which limits the utility of doing the unit testing. (This was the impetus for asking the question.) There also seems to be some potential trickiness in calling the unit-testing framework.
- Write output to file (in PETSc, for instance, using
VecView
andMatView
), and compare against known output with something likendiff
ornumdiff
. My gut feeling with this method from previous experience doing unit testing with file comparisons is that it will be finicky, and it will require some filtering. This method seems like it would be excellent for regression testing, though, because I could replace the utilities above with a plaindiff
, and not have to worry about matching up text formats. I've gathered that this strategy is more or less what WolfgangBangerth and andybauer are suggesting. PETSc also appears to use a similar approach for some of the testing it does. - Use a unit testing framework, gather everything onto the processor with MPI rank 0, and ask it to execute unit tests only if the processor rank is 0. I could do something similar with norms (it's probably even easier that way), although the tradeoff is that any errors returned will tell me that I have a problem in my calculation, but not which elements are in error. Then I don't need to worry about any unit testing output being garbled; I only need to worry about calling the unit testing framework correctly. PETSc appears to use norm-wise comparisons within its example programs when exact solutions are available, but it does not use a unit testing framework when making those comparisons (nor should it, necessarily).
mpiexec
to run it, and include calls likePETScInitialize
/PETScFinalize
in the setup/teardown code. (Presumably, if I weren't using PETSc, I'd replace those calls with analogues ofMPI_Init
/MPI_Finalize
, depending on the libraries I'm using.) Google's testing framework is a source-based release, so compiling it along with the code I write also would not be a problem. $\endgroup$RHSFunction
andRHSJacobian
in${PETSC_DIR}/src/ts/examples/tutorials/ex.2
) in isolation. $\endgroup$