4
$\begingroup$

In light of this question and some stuff I read online I am wondering if large FE libs (e.g., deal.ii, libmesh etc.) that do AMR keep the entire mesh or possibly a coarse version of the entire mesh on each core. Also, does this limit the size of the problems that can be solved on machines with low RAM per core.

Finally how is load balancing performed? Is the entire mesh repartitioned after each refinement/coarsening step?

$\endgroup$

1 Answer 1

8
$\begingroup$

deal.II keeps the entire coarse mesh on every processor, but of course we cannot do that with the actual mesh after many refinement steps. It is true that this somewhat limits the size of problems we can solve to maybe a few hundred thousand coarse mesh cells on typically-sized cluster nodes. However, this is plenty for most realistic cases for a reasonable coarse approximation of the geometry. Adaptive mesh refinement can then resolve the details -- and for the refined mesh you can go about as far as you want, we have done meshes with several billion cells.

As far as load balancing is concerned, yes, we do that after every refinement/coarsening step. It turns out that this is not an operation that requires a noticeable fraction of the overall run time of programs, even on large numbers of processors.

I admit that I do not know whether the observations above are true for other libraries. The authors of these libraries need to speak for their own products.

$\endgroup$
2
  • $\begingroup$ Thanks for clarifying. I guess one can always decrease the number of cores used per node if memory/core does become an issue. Do you use parmetis/ptscotch with deal.ii? Also, does the partitioner at any step use arrays/vectors that are proportional to the problem size. I know metis (serial) needs to have info about the entire mesh but I am not sure about parmetis/ptscotch. $\endgroup$
    – stali
    Commented Mar 5, 2014 at 14:50
  • $\begingroup$ We use the p4est library for partitioning the parallel mesh. And no, you can't keep any vectors that are proportional to the problem size, though there are a few arrays we need to keep with size equal to the number of MPI processes. As long as our jobs are only a couple of 10,000 cores, this is not too much of a problem. It will be a problem when we get to $10^7$ or more cores. $\endgroup$ Commented Mar 5, 2014 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.