Any recommendations for unit-testing frameworks compatible with code/libraries that use MPI?

Usually, I write serial code, and when I do, I write unit tests with some xUnit-style testing framework (MATLAB xUnit, PyUnit/nose, or Google's C++ testing framework).

Based on a cursory Google search, I haven't seen much on how practitioners unit test code that uses MPI. Are there any best practices for that?

Compared to Strategies for unit testing and test-driven development, I'm looking for answers relating to what software I should use for a testing framework (if any exist -- the answer could very well be "roll your own code", in which case examples of custom testing code would be helpful).

Most of what I'm looking to test are right-hand side function evaluations and Jacobian matrix assembly routines for time steppers that will be integrating semi-discretized PDEs. I will be using PETSc, so if there's anything PETSc-specific, that would be helpful in addition to more general testing frameworks.

Clarification Edits:

An example would be in ${PETSC_DIR}/src/ts/examples/tutorials/ex2.c, where I would want to test something like RHSFunction (a right-hand side function evaluation) and RHSJacobian (a Jacobian matrix evaluation). I'd be testing against known values for the assembled right-hand side and assembled Jacobian matrix; I can obtain these values analytically for some simple problem instances. These functions are application-specific functions that won't exercise any other application-level function, but they could call MPI if vector or matrix assembly is done within the function (as in the linked PETSc example above). If I write functions that only calculate portions of vectors or matrices local to a processor, I would want to test against the global, assembled version if possible because, being new to parallel programming, it's more intuitive for me to think of global vectors and global matrices. These tests would be run on small problem sizes and small numbers of processors. I can think of a few strategies to do this: • A strategy that probably won't work well, based on the Google searches I've done on this topic, would be to construct a known output, find the relative/absolute error in parallel, and then do naive comparisons. The output will probably be garbled -- anyone who has written a "Hello, world" program with MPI knows why -- which limits the utility of doing the unit testing. (This was the impetus for asking the question.) There also seems to be some potential trickiness in calling the unit-testing framework. • Write output to file (in PETSc, for instance, using VecView and MatView), and compare against known output with something like ndiff or numdiff. My gut feeling with this method from previous experience doing unit testing with file comparisons is that it will be finicky, and it will require some filtering. This method seems like it would be excellent for regression testing, though, because I could replace the utilities above with a plain diff, and not have to worry about matching up text formats. I've gathered that this strategy is more or less what WolfgangBangerth and andybauer are suggesting. PETSc also appears to use a similar approach for some of the testing it does. • Use a unit testing framework, gather everything onto the processor with MPI rank 0, and ask it to execute unit tests only if the processor rank is 0. I could do something similar with norms (it's probably even easier that way), although the tradeoff is that any errors returned will tell me that I have a problem in my calculation, but not which elements are in error. Then I don't need to worry about any unit testing output being garbled; I only need to worry about calling the unit testing framework correctly. PETSc appears to use norm-wise comparisons within its example programs when exact solutions are available, but it does not use a unit testing framework when making those comparisons (nor should it, necessarily). • I'm only familiar with in-house testing suites, so I can't recommend anything. That being said, do none of these testing suites allow you to specify how to run the executable you create? If they do, it ought to be trivial to build tests that work for MPI programs. – Bill Barth Sep 17 '13 at 20:59 • They should. In any compiled language, it's just an executable, so it should be no problem to use mpiexec to run it, and include calls like PETScInitialize/PETScFinalize in the setup/teardown code. (Presumably, if I weren't using PETSc, I'd replace those calls with analogues of MPI_Init/MPI_Finalize, depending on the libraries I'm using.) Google's testing framework is a source-based release, so compiling it along with the code I write also would not be a problem. – Geoff Oxberry Sep 17 '13 at 21:12 • Your description of the problem suggests to me that you're interested in using a unit-testing framework to run integration/regression tests. There's nothing wrong with that per se, but you might want to clarify your question a bit more. I think if you asked a unit testing expert how to write unit tests for your scientific code, they'd tell you to write tests in a modular way. That is, most of your tests wouldn't have proper MPI calls in them. – Aron Ahmadia Sep 19 '13 at 16:15 • Let me be more concrete. Something I would want to test on a small problem with a small number of processors (say, 1-4) would be whether or not my assembled Jacobian matrix actually results in the proper global Jacobian. I would also want to test my right-hand side function against a known global right-hand side. Each such test should still only exercise a single function in the application (for instance, in PETSc, testing RHSFunction and RHSJacobian in ${PETSC_DIR}/src/ts/examples/tutorials/ex.2) in isolation. – Geoff Oxberry Sep 19 '13 at 17:49
• I don't think a framework currently exists that is going to help you do what you want. We've managed to wrangle nose into doing a few things for us in PyClaw, (and Lisandro has used it in mpi4py and petsc4py). Have you looked at the testing framework in mpich? – Aron Ahmadia Sep 20 '13 at 1:48

I'm a happy user of GoogleTest with a C++ MPI code in a CMake/CTest build environment:

• adding tests is a one-liner!
• writing the tests is easy! (and google mock is very powerful!)
• CTest can pass command-line parameters to your tests, and exports data to CDash!

This is how it works. A batch of unit-tests that require mpi are written into some my_mpi_test.cpp file which looks like this:

#include <gtest/gtest.h>
#include <boost/mpi.h>

/// Most testing libraries allow to define main yourself to override initialization.
int main(int argc, char* argv[]) {
::testing::InitGoogleTest(&argc, argv);  /// Set gtest environment
mpi::environment env(argc, argv);  /// Set mpi environment
return RUN_ALL_TESTS();  /// Execute all gtest tests
}

TEST(test_batch_name, test_name) {  /// Then you can create tests as usual,
using namespace mpi;
communicator world;  /// and use MPI inside your tests.
/* ... test stuff here ... */
}


The CMakeLists.txt that adds this test is:

add_mpi_test(my_mpi 2)  # Uses 2 MPI processes


where add_mpi_test wraps CMake's add_test inside my root CMakeLists.txt:

function(add_mpi_test name no_mpi_proc)
include_directories(${MY_TESTING_INCLUDES}) # My test are all called name_test.cpp add_executable(${name} ${name}_test.cpp) add_dependencies(${name} googletest)
# Make sure to link MPI here too:
target_link_libraries(${name}${MY_TESTING_LIBS})
set(test_parameters ${MPIEXEC_NUMPROC_FLAG}${no_mpi_proc} "./${name}") add_test(NAME${name} COMMAND ${MPIEXEC}${test_parameters})


This last part is not necessary but allows you to easily add mpi tests in one line. Then you can decide if you want to hard-code the numer of MPI processes for each test or read it via a command line parameter to ctest.

We simply roll our own code in deal.II -- in essence, we tell the framework to execute tests using mpirun -np .... We had previously just used a Makefile-based testing scheme (compile, link, execute test, then compare the output with one that had previously been saved) and you can find this here:

and for context, the non-MPI targets are here:

We are rewriting things using CMake/CTest, with the current development here:

• Wolfgang, thanks for the answer! PETSc seems to do something similar. – Geoff Oxberry Sep 19 '13 at 6:12

There are several MPI-enabled software packages that use the CMake set of tools for testing. The ones that I can think of off the top of my head are Trilinos, VTK and ParaView. I would think that you don't want to assume that the executable needs to be launched with mpirun and/or mpiexec. CMake has support for specifying how to properly launch the executable along with different options such as the maximum number of processes to use and pre- and post-flags, if needed.

You may want to look at the HPC Sites section of the ParaView dashboard where the tests are run on a variety of NERSC and Argonne supercomputers. Buried in there are also most of the settings that you'd need to specify in order to get it working on those machines.

For reference, the Trilinos dashboard has a wide variety of packages listed and to me is rather impressive in its organization.

Full disclosure: I am a Kitware employee and CMake is one of the open source projects that Kitware is involved with.

• Thanks for the answer! I've been looking at CTest, and haven't come across any documentation aside from a man-page-like description on the KitWare web site. Can you recommend any freely available tutorials? – Geoff Oxberry Sep 19 '13 at 6:14
• There's a bunch of information on the CMake wiki. There are a bunch of tutorials for CMake, CTest and CPack there. I find most of my answers to those applications on Stack Overflow. – andybauer Sep 19 '13 at 14:42