# Overlapping communication and computation in PETSc and/or Trilinos?

I am just learning about these packages, so forgive me if this is a trivial/silly question.

Our group is working re-developing our code from the ground up using modern software practices. Currently it is an explicit wave propagation code for earthquake source physics, but more and more our group is working on other coupled seismic source problems. In the future we will need to ability to handle portions of the domain implicitly, use unstructured grids, and (like many others in the numerical PDE world) will consider using GPUs. Thus we really want to make the right design decisions to facilitate this from the get go.

One of the big questions were are looking at is which of the big national lab packages to develop with in mind, PETSc or Trilinos. One of the question that I have been seeking an answer to is how easy it is to overlap computation, I/O, and communication with these packages. For instance, if we used the distributed array data structures of PETSc does this means that we will not be able to exchange our halo cells while applying our stencil to the rest of the domain? (Similarly with the EPetra/TPetra types of Trilinos.)

-

Yes, many communication functions are split into begin/end pairs. A typical example is the code for MatMult_MPIAIJ() (the standard row-based format).

VecScatterBegin(a->Mvctx,xx,a->lvec,INSERT_VALUES,SCATTER_FORWARD);
(*a->A->ops->mult)(a->A,xx,yy);            /* multiply the diagonal block */
VecScatterEnd(a->Mvctx,xx,a->lvec,INSERT_VALUES,SCATTER_FORWARD);

-
Is this a fairly new structure? Wondering, b/c this (originally from 2010) is what originally lead my astray when googling –  Jeremy Kozdon Feb 24 '12 at 2:52
No, this is very old support. That user was asking for a very specific granularity that PETSc don't specifically support. That particular suggestion is usually slower because the memory access has worse locality and because it has more communication phases allowing for more skew due to minor load imbalances or system/network noise. –  Jed Brown Feb 24 '12 at 3:03
Great. Thanks. This is all new to me, I appreciate your help. –  Jeremy Kozdon Feb 24 '12 at 5:54

Clearly, I need to learn how to search better!

In PETSc this can be accomplished (I believe) with (pg 49 of the users manual)

   DMGlobalToLocalBegin(DM da,Vec g,InsertMode iora,Vec l);
DMGlobalToLocalEnd(DM da,Vec g,InsertMode iora,Vec l);

-