Halo Region Communication in Unstructured Mesh Problems

Question

I'm currently using ParMETIS and it is required to determine the halo region of the local elements in a parallel unstructured mesh. Assume that the mesh is large and cannot be stored on a single processor.

My approach is to gather the global list of elements and their processor location in each processor. Then the neighbors of the local elements which do not belong to the current processor are identified and are then communicated.

There is redundant communication being performed as some of the non-local elements are clearly not neighbors to any of the local elements and are hence not going to be in the halo region, but their location is requested.

What is the most efficient procedure to communicate halo information in unstructured mesh calculations? (assuming communication paradigm is MPI and calls from ParMETIS are available for obtaining dual graphs, global adjacency and so on)

Answer 1

The best to see what others do.

You need so-called ghost nodes and ghost cells. The best is to look how ghost vectors are implemented in PETSc; you can find in the documentation how it works there.

In principle, parallel vector keeps a local copy of unknowns on each processor, also, also has extra ghost data at the end, which are used for communication.

You have to start which nodes you own, which nodes are shared and which of them are ghosted. Typically which are in overlapped region nodes are owned by a lower processor, but you can have more strategies.

PETSc manages algebra (and mesh see DMPLEX) if you like to see how this is managed on mesh sied, see MOAB, http://sigma.mcs.anl.gov/moab-library/ look at the documentation.

Programming this you would like to use a number of communication events if you do communication try to do all at once, MoAB has class CrystalRouter for this, see http://ftp.mcs.anl.gov/pub/fathom/moab-docs/

I ma not sure if this is what you are looking for.

Answer 2

The first thing is to find the halo cells of the current processor and its owner-processor. We create a list to record it. Then we use the MPI_SEND and MPI_receive to conduct the communication.