Overlapping communication and computation in PETSc and/or Trilinos?

Question

I am just learning about these packages, so forgive me if this is a trivial/silly question.

Our group is working re-developing our code from the ground up using modern software practices. Currently it is an explicit wave propagation code for earthquake source physics, but more and more our group is working on other coupled seismic source problems. In the future we will need to ability to handle portions of the domain implicitly, use unstructured grids, and (like many others in the numerical PDE world) will consider using GPUs. Thus we really want to make the right design decisions to facilitate this from the get go.

One of the big questions were are looking at is which of the big national lab packages to develop with in mind, PETSc or Trilinos. One of the question that I have been seeking an answer to is how easy it is to overlap computation, I/O, and communication with these packages. For instance, if we used the distributed array data structures of PETSc does this means that we will not be able to exchange our halo cells while applying our stencil to the rest of the domain? (Similarly with the EPetra/TPetra types of Trilinos.)

Answer 1

Yes, many communication functions are split into begin/end pairs. A typical example is the code for MatMult_MPIAIJ() (the standard row-based format).

VecScatterBegin(a->Mvctx,xx,a->lvec,INSERT_VALUES,SCATTER_FORWARD);
(*a->A->ops->mult)(a->A,xx,yy);            /* multiply the diagonal block */
VecScatterEnd(a->Mvctx,xx,a->lvec,INSERT_VALUES,SCATTER_FORWARD);
(*a->B->ops->multadd)(a->B,a->lvec,yy,yy); /* add contribution from off-diagonal block */

Answer 2

Clearly, I need to learn how to search better!

In PETSc this can be accomplished (I believe) with (pg 49 of the users manual)

   DMGlobalToLocalBegin(DM da,Vec g,InsertMode iora,Vec l);
   DMGlobalToLocalEnd(DM da,Vec g,InsertMode iora,Vec l);

Answer 3

I started out as a Trilinos guy (mostly through the Python bindings), and wrote a big fluids code that worked with both it and PETSc. In general, I would say that these are matrix libraries, and the communications functions are cludgy and somewhat annoying. TPetra from the Trilinos folks looks really rad, but seems to cause nothing but heartache on the trilinos-users listserv. . .

I would recommend PETSc personally, it seems really active in development and use. The one caveat I would offer is that if you need to pass something more complex than a vector, using a different channel for communication such as Zero-MQ can really ease things.