I am just learning about these packages, so forgive me if this is a trivial/silly question.
Our group is working re-developing our code from the ground up using modern software practices. Currently it is an explicit wave propagation code for earthquake source physics, but more and more our group is working on other coupled seismic source problems. In the future we will need to ability to handle portions of the domain implicitly, use unstructured grids, and (like many others in the numerical PDE world) will consider using GPUs. Thus we really want to make the right design decisions to facilitate this from the get go.
One of the big questions were are looking at is which of the big national lab packages to develop with in mind, PETSc or Trilinos. One of the question that I have been seeking an answer to is how easy it is to overlap computation, I/O, and communication with these packages. For instance, if we used the distributed array data structures of PETSc does this means that we will not be able to exchange our halo cells while applying our stencil to the rest of the domain? (Similarly with the EPetra/TPetra types of Trilinos.)