Using same memory space for global and local Petsc DA Vectors

Question

I have constructed a local and global distributed array in Petsc 3.2 using:

DMCreateGlobalVector(da, &v);
DMCreateLocalVector(da, &lv);

In order to retrieve the ghost cell values from neighbouring processes, I simply use the command:

DMGlobalToLocalBegin(da, v, INSERT_VALUES, lv);
DMGlobalToLocalEnd(da, v, INSERT_VALUES, lv);

From what I understand, the local and global vectors use separate memory spaces. For any processor, the local portion of the global vector (Vec v) is duplicated in another memory space allocated to the local vector (Vec lv). I am attempting to construct a local/global vector that use the same memory space. I would then need to use some sort of scatter routine to only fill the appropriate ghost cells. I notice that there is a VecCreateGhost function that seems to provide this functionality. However, it does not seem to be associated with the DMDA functions (e.g. DMCreateGhostVector).

Answer 1

If you have a structured grid then this is unquestionably premature optimization. Direct access in more than one dimension requires that some of the ghost values are "between" owned values, thus a copy is required.

If you use Krylov methods, accelerated nonlinear methods (NGMRES), or multistage time integrators, then at the end of the day use of VecGhost in place of normal global vectors will almost always require more memory because the extra space is stored as part of every vector.

So unless you have a very unique application, you really shouldn't try to use VecGhost for structured grids. For unstructured grids, it's arguably a natural representation and typically not harmful unless you have larger stencils, but it's also not a performance optimization. If you have have a very special use case, you can implement DMCreateGlobalVector_XXX() to return a VecGhost.