I am just learning about these packages, so forgive me if this is a trivial/silly question.

Our group is working re-developing our code from the ground up using modern software practices. Currently it is an explicit wave propagation code for earthquake source physics, but more and more our group is working on other coupled seismic source problems. In the future we will need to ability to handle portions of the domain implicitly, use unstructured grids, and (like many others in the numerical PDE world) will consider using GPUs. Thus we really want to make the right design decisions to facilitate this from the get go.

One of the big questions were are looking at is which of the big national lab packages to develop with in mind, PETSc or Trilinos. One of the question that I have been seeking an answer to is how easy it is to overlap computation, I/O, and communication with these packages. For instance, if we used the distributed array data structures of PETSc does this means that we will not be able to exchange our halo cells while applying our stencil to the rest of the domain? (Similarly with the EPetra/TPetra types of Trilinos.)


Yes, many communication functions are split into begin/end pairs. A typical example is the code for MatMult_MPIAIJ() (the standard row-based format).

(*a->A->ops->mult)(a->A,xx,yy);            /* multiply the diagonal block */
(*a->B->ops->multadd)(a->B,a->lvec,yy,yy); /* add contribution from off-diagonal block */
  • $\begingroup$ Is this a fairly new structure? Wondering, b/c this (originally from 2010) is what originally lead my astray when googling $\endgroup$ – Jeremy Kozdon Feb 24 '12 at 2:52
  • $\begingroup$ No, this is very old support. That user was asking for a very specific granularity that PETSc don't specifically support. That particular suggestion is usually slower because the memory access has worse locality and because it has more communication phases allowing for more skew due to minor load imbalances or system/network noise. $\endgroup$ – Jed Brown Feb 24 '12 at 3:03
  • $\begingroup$ Great. Thanks. This is all new to me, I appreciate your help. $\endgroup$ – Jeremy Kozdon Feb 24 '12 at 5:54

Clearly, I need to learn how to search better!

In PETSc this can be accomplished (I believe) with (pg 49 of the users manual)

   DMGlobalToLocalBegin(DM da,Vec g,InsertMode iora,Vec l);
   DMGlobalToLocalEnd(DM da,Vec g,InsertMode iora,Vec l);

I started out as a Trilinos guy (mostly through the Python bindings), and wrote a big fluids code that worked with both it and PETSc. In general, I would say that these are matrix libraries, and the communications functions are cludgy and somewhat annoying. TPetra from the Trilinos folks looks really rad, but seems to cause nothing but heartache on the trilinos-users listserv. . .

I would recommend PETSc personally, it seems really active in development and use. The one caveat I would offer is that if you need to pass something more complex than a vector, using a different channel for communication such as Zero-MQ can really ease things.

  • $\begingroup$ Interesting. Yeah, I am having a hard time deciding between these two packages since both are more than we need right now, and both have a steep learning curve. Our code is currently a high-order, multiblock finite difference code. But we have visions of extending it to include unstructured FVM blocks and blocks with different physics. We don't currently need to solve large linear systems, but will in the near future. And everything has to be parallel. $\endgroup$ – Jeremy Kozdon Feb 28 '12 at 20:12
  • $\begingroup$ Here is link to my biggest piece of Trilinos code It will give you some idea for the annoyance one must go through to make a parallel code. It worked great over 500 nodes though for a couple of really big solves! $\endgroup$ – meawoppl Feb 29 '12 at 5:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.