Any recommendations for unit-testing frameworks compatible with code/libraries that use MPI?

Question

Usually, I write serial code, and when I do, I write unit tests with some xUnit-style testing framework (MATLAB xUnit, PyUnit/nose, or Google's C++ testing framework).

Based on a cursory Google search, I haven't seen much on how practitioners unit test code that uses MPI. Are there any best practices for that?

Compared to Strategies for unit testing and test-driven development, I'm looking for answers relating to what software I should use for a testing framework (if any exist -- the answer could very well be "roll your own code", in which case examples of custom testing code would be helpful).

Most of what I'm looking to test are right-hand side function evaluations and Jacobian matrix assembly routines for time steppers that will be integrating semi-discretized PDEs. I will be using PETSc, so if there's anything PETSc-specific, that would be helpful in addition to more general testing frameworks.

Clarification Edits:

An example would be in ${PETSC_DIR}/src/ts/examples/tutorials/ex2.c, where I would want to test something like RHSFunction (a right-hand side function evaluation) and RHSJacobian (a Jacobian matrix evaluation). I'd be testing against known values for the assembled right-hand side and assembled Jacobian matrix; I can obtain these values analytically for some simple problem instances. These functions are application-specific functions that won't exercise any other application-level function, but they could call MPI if vector or matrix assembly is done within the function (as in the linked PETSc example above). If I write functions that only calculate portions of vectors or matrices local to a processor, I would want to test against the global, assembled version if possible because, being new to parallel programming, it's more intuitive for me to think of global vectors and global matrices. These tests would be run on small problem sizes and small numbers of processors.

I can think of a few strategies to do this:

• A strategy that probably won't work well, based on the Google searches I've done on this topic, would be to construct a known output, find the relative/absolute error in parallel, and then do naive comparisons. The output will probably be garbled -- anyone who has written a "Hello, world" program with MPI knows why -- which limits the utility of doing the unit testing. (This was the impetus for asking the question.) There also seems to be some potential trickiness in calling the unit-testing framework.
• Write output to file (in PETSc, for instance, using VecView and MatView), and compare against known output with something like ndiff or numdiff. My gut feeling with this method from previous experience doing unit testing with file comparisons is that it will be finicky, and it will require some filtering. This method seems like it would be excellent for regression testing, though, because I could replace the utilities above with a plain diff, and not have to worry about matching up text formats. I've gathered that this strategy is more or less what WolfgangBangerth and andybauer are suggesting. PETSc also appears to use a similar approach for some of the testing it does.
• Use a unit testing framework, gather everything onto the processor with MPI rank 0, and ask it to execute unit tests only if the processor rank is 0. I could do something similar with norms (it's probably even easier that way), although the tradeoff is that any errors returned will tell me that I have a problem in my calculation, but not which elements are in error. Then I don't need to worry about any unit testing output being garbled; I only need to worry about calling the unit testing framework correctly. PETSc appears to use norm-wise comparisons within its example programs when exact solutions are available, but it does not use a unit testing framework when making those comparisons (nor should it, necessarily).

Answer 1

I'm a happy user of GoogleTest with a C++ MPI code in a CMake/CTest build environment:

• CMake automatically installs/links googletest from svn!
• adding tests is a one-liner!
• writing the tests is easy! (and google mock is very powerful!)
• CTest can pass command-line parameters to your tests, and exports data to CDash!

This is how it works. A batch of unit-tests that require mpi are written into some my_mpi_test.cpp file which looks like this:

#include <gtest/gtest.h>
#include <boost/mpi.h>

/// Most testing libraries allow to define main yourself to override initialization.
int main(int argc, char* argv[]) {
    ::testing::InitGoogleTest(&argc, argv);  /// Set gtest environment
    mpi::environment env(argc, argv);  /// Set mpi environment
    return RUN_ALL_TESTS();  /// Execute all gtest tests
}

TEST(test_batch_name, test_name) {  /// Then you can create tests as usual,
  using namespace mpi;
  communicator world;  /// and use MPI inside your tests.
  /* ... test stuff here ... */
}

The CMakeLists.txt that adds this test is:

add_mpi_test(my_mpi 2)  # Uses 2 MPI processes

where add_mpi_test wraps CMake's add_test inside my root CMakeLists.txt:

function(add_mpi_test name no_mpi_proc)
  include_directories(${MY_TESTING_INCLUDES})
      # My test are all called name_test.cpp
      add_executable(${name} ${name}_test.cpp)
      add_dependencies(${name} googletest)
  # Make sure to link MPI here too:
  target_link_libraries(${name} ${MY_TESTING_LIBS})
  set(test_parameters ${MPIEXEC_NUMPROC_FLAG} ${no_mpi_proc} "./${name}")
      add_test(NAME ${name} COMMAND ${MPIEXEC} ${test_parameters})
endfunction(add_mpi_test)

This last part is not necessary but allows you to easily add mpi tests in one line. Then you can decide if you want to hard-code the numer of MPI processes for each test or read it via a command line parameter to ctest.

Answer 2

We simply roll our own code in deal.II -- in essence, we tell the framework to execute tests using mpirun -np .... We had previously just used a Makefile-based testing scheme (compile, link, execute test, then compare the output with one that had previously been saved) and you can find this here:

• https://svn.dealii.org/branches/releases/Branch-8-0/tests/mpi/Makefile

and for context, the non-MPI targets are here:

• https://svn.dealii.org/branches/releases/Branch-8-0/tests/Makefile.rules

We are rewriting things using CMake/CTest, with the current development here:

• https://svn.dealii.org/branches/branch_port_the_testsuite/tests/CMakeLists.txt
• https://svn.dealii.org/branches/branch_port_the_testsuite/tests/cmake/

Answer 3

There are several MPI-enabled software packages that use the CMake set of tools for testing. The ones that I can think of off the top of my head are Trilinos, VTK and ParaView. I would think that you don't want to assume that the executable needs to be launched with mpirun and/or mpiexec. CMake has support for specifying how to properly launch the executable along with different options such as the maximum number of processes to use and pre- and post-flags, if needed.

You may want to look at the HPC Sites section of the ParaView dashboard where the tests are run on a variety of NERSC and Argonne supercomputers. Buried in there are also most of the settings that you'd need to specify in order to get it working on those machines.

For reference, the Trilinos dashboard has a wide variety of packages listed and to me is rather impressive in its organization.

Full disclosure: I am a Kitware employee and CMake is one of the open source projects that Kitware is involved with.

Answer 4

The Teuchos Unit test harness in Trilinos natively supports unit tests that use MPI. Things like controlling output from multiple processes and aggregating pass/fail over all processes is automatic. Take a look:

http://trilinos.org/docs/dev/packages/teuchos/doc/html/group__Teuchos__UnitTest__grp.html