4
$\begingroup$

Why does DMDA objects require so much memory on PETSc 3.2-p7.

When running the code:

  ...
  N = 8000;
  ierr = DMDACreate2d(PETSC_COMM_WORLD, DMDA_BOUNDARY_PERIODIC, DMDA_BOUNDARY_PERIODIC, 
            DMDA_STENCIL_STAR, N, N, size, PETSC_DECIDE, PETSC_DECIDE, 
            0, PETSC_NULL, PETSC_NULL, &da);
  //ierr = DMCreateGlobalVector(da, &x); CHKERRQ(ierr);
  ...

using the command ./prog -log_summary, I get the following memory usage summary:

   Distributed Mesh: 512004856 bytes
   Vector: 2944 bytes
   Vector Scatter: 1256 bytes
   Index Set: 256003752 bytes
   IS L to G Mapping: 256000572 bytes

By uncommenting the DMCreateGlobalVector line, the vector size in the memory summary increases to:

   Vector: 512004384

It appears that DMDA objects have a size proportional to the vector data size. According to the documentation, the data is not stored in the Distributed Mesh. Therefore, why does the DMDA object require this much memory? Is there a way to reduce the memory overhead used by DMDA?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

That storage is in the scatters and local-to-global maps. The DMDA also creates (and destroys) a vector during setup; it likely doesn't count toward peak memory usage.

Do you have an application where this limits your capability in any way or is it an academic interest? There are normally a few vectors in a simulation (and for some applications, a matrix) so the DMDA memory usage is fairly insignificant and worth it in exchange for the faster data structures.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I was investigating the performance of some preliminary code. I wanted to get an idea of the memory usage, before I ran it on a cluster. The program used twice as much memory than I was expecting. I just wanted to find an explanation. When running the program above (including the DMCreateGlobalVector line) through Valgrind (Massif), the program uses 978.70 MB of heap memory when there is only 488.28 MB worth of data. When DMDA destroys the vector on setup, is the memory freed? Or is the memory kept for future use? $\endgroup$
    – Eldila
    Jun 10, 2012 at 6:28
  • $\begingroup$ Using Massif, it appears that the "unexpected" memory usage came from the local-to-global maps. Both DMSetUp_DA_2D (da2.c:1795) and ISLocalToGlobalMappingCreate (isltog.c:165) allocates an INT for every local vertex in the DA. This would explain why my memory usage was double what I expected. I assume there is no way around this. $\endgroup$
    – Eldila
    Jun 10, 2012 at 6:53
  • $\begingroup$ PETSc itself assumes the data structure is static, but it wouldn't be out of the question to design an interface that doesn't rely on the existence of the mapping for communication by computing the indices on the fly. In the current versions of MPI you would likely need this indices in memory anyway, so it might not be particularly useful to try this before MPI-3. I think this is a separate question, if you'd like to ask it. $\endgroup$
    – Aron Ahmadia
    Jun 10, 2012 at 7:34
  • $\begingroup$ It would be possible to implement packing and unpacking without storing those integers, but it would be messier and break some abstractions that enable faster code. Are you actually doing a computation that only ever stores one vector? Most computations need at least a local vector (to hold state with ghost points) and a global vector (to compute norms). (And usually a residual and a defect correction; perhaps also stages/old steps and an error estimate. And we've hardly started with implicit solvers.) Are you sure you will only use one vector (especially a global one)? $\endgroup$
    – Jed Brown
    Jun 10, 2012 at 13:13
  • $\begingroup$ I will be using multiple vectors with ghost cells and compute norms. However, I won't be needing some of the higher level packages: KSP, TS, SNES, etc. I was using PETSc to provide an abstraction when working with 2d/3d scalar fields. I understand why the maps are needed for unstructured grids, but I was surprised to find them in DMDAs. They could be computed on the fly as far as I could tell. The situation is not as bad as I originally thought though. When using the -log_summary, the memory summary included the size of the temporary vector. This gave the impression that things were worse. $\endgroup$
    – Eldila
    Jun 10, 2012 at 15:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.