I am currently developing a domain decomposition method for the solution of the scattering problem. Basically I am solving a system of Helmholtz BVPs iteratively. I discretize the equations using finite element method over triangular or tetrahedral meshes. I'm developing the code towards my Phd thesis. I am aware of some of the existing finite element libraries out there such as deal.ii or DUNE and though I think they are great, with inspirational design and API, for learning purposes I wanted to develop my own little application from scratch.
I am at a point where I have my serial versions running and now I want to parallelize them. After all, it is one of the strengths of the domain decomposition framework to formulate algorithms that are easy to parallelize, at least in principle. In practice however, there are many details one must consider. Mesh management is one of them. If the applications is to achieve high resolution while scaling well to many CPUs the replication of an entire mesh on every CPU is inefficient.
I wanted to ask those developers who work on similar applications in high performance computing environments how they deal with this issue.
There is p4est library for distributed mesh management. I do not need AMR so it might be an overkill since I'm only interested in using uniform meshes and I'm not sure if it can refine triangular meshes. I could also simply create a uniform mesh then feed it into one of the mesh partitioners and do some post processing of the output.
The simplest approach seems to create a separate file for each partition containing mesh information relevant only to that particular partition. This file would be read by a single CPU which would be responsible for assembly of the discrete system on that portion of the mesh. Of course, some global partition connectivity/neighborhood information would also need to be stored in a file read by all CPUs for inter process communication.
What other approaches are out there? If some of you could share, what are some of the commonly used methodologies in the industry, or government research institutions related to handling this issue? I am quite new to programming a parallel finite element solver and I wanted to get a feel for whether or not I'm thinking about this problem correctly and how others are approaching it. Any advice or pointers to relevant research articles would be greatly appreciated!
Thanks in advance!