I have constructed a local and global distributed array in Petsc 3.2 using:

DMCreateGlobalVector(da, &v);
DMCreateLocalVector(da, &lv);

In order to retrieve the ghost cell values from neighbouring processes, I simply use the command:

DMGlobalToLocalBegin(da, v, INSERT_VALUES, lv);
DMGlobalToLocalEnd(da, v, INSERT_VALUES, lv);

From what I understand, the local and global vectors use separate memory spaces. For any processor, the local portion of the global vector (Vec v) is duplicated in another memory space allocated to the local vector (Vec lv). I am attempting to construct a local/global vector that use the same memory space. I would then need to use some sort of scatter routine to only fill the appropriate ghost cells. I notice that there is a VecCreateGhost function that seems to provide this functionality. However, it does not seem to be associated with the DMDA functions (e.g. DMCreateGhostVector).


1 Answer 1


If you have a structured grid then this is unquestionably premature optimization. Direct access in more than one dimension requires that some of the ghost values are "between" owned values, thus a copy is required.

If you use Krylov methods, accelerated nonlinear methods (NGMRES), or multistage time integrators, then at the end of the day use of VecGhost in place of normal global vectors will almost always require more memory because the extra space is stored as part of every vector.

So unless you have a very unique application, you really shouldn't try to use VecGhost for structured grids. For unstructured grids, it's arguably a natural representation and typically not harmful unless you have larger stencils, but it's also not a performance optimization. If you have have a very special use case, you can implement DMCreateGlobalVector_XXX() to return a VecGhost.

  • $\begingroup$ I will get back to you if this becomes a problem. I am still in the middle of development. I thought there might be an easy way to reduce the memory requirements of my application by using more advanced features of petsc. Thanks for your input. $\endgroup$
    – Eldila
    Sep 11, 2012 at 15:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.