# Parallelisation strategies for mixed FE formulations

Mixed FE formulations with LBB-stable elements require two different meshes for the primary and the constraint variables, for example, displacement and pressure. With continuous approximation for the pressure field, I am finding it difficult to parallelise for distributed memory architectures.

I am interested in learning some commonly employed parallelisation strategies for such problems. I very much appreciate any useful resources on this topic.

Note that I use the PETSc library for solving the matrix system in my C++ code.

• Are the meshes for both fields same but just the polynomial degree are different? For example, do you use a triangular mesh for displacement and a rectangular mesh for pressure? Is it such a case? – Abdullah Ali Sivas Aug 24 '20 at 20:18
• The mesh is the same but with different orders of polynomials for different fields, like the Taylor-Hood elements, P2/P1 and Q2/Q1. – Chenna K Aug 24 '20 at 23:05
• Then there are many ways to handle it, Wolfgang Bangerth is one of the developers of deal.ii so consider his answer and advise. But I am also very fond of the way MFEM handles it (mfem.org/performance) – Abdullah Ali Sivas Aug 24 '20 at 23:20
• Thank you for the link! MFEM is a great library. I will go through the documentation. – Chenna K Aug 24 '20 at 23:27

## 2 Answers

It's a misunderstanding that you need two different meshes: The proper way to see things is that you are using the same mesh, but different polynomial spaces for the two variables. For example, for the Stokes equation, you'd have quadratic polynomials for the velocity $$\mathbf u$$ and linear polynomials for the pressure $$p$$.

Appropriate parallelization strategies are then to partition the mesh among processors. This also induces a partitioning of degrees of freedom, and consequently of those rows of the matrix (and vector elements) each processor stores. It's really no different than if you had a scalar problem.

• I was thinking the same but then there are inf-sup stable finite elements where the velocity mesh is simplectic and the pressure mesh is tensor-product. I wonder if it is such a case. – Abdullah Ali Sivas Aug 24 '20 at 20:17
• Thank you @Wolfgang! I agree that we don't need two different meshes. With a single mesh, one can get processor IDs for each element and node in the mesh using METIS. However, processor id colouring is available only for either displacement nodes or pressure nodes. The straightforward technique would be to use the same colouring for pressure nodes as that of displacement nodes. This is what I am currently implementing. – Chenna K Aug 24 '20 at 22:55
• @ChennaK: Well, you should color cells and then infer the color of nodes based on that of the cells. Then you have the same partitioning for pressures and velocities. – Wolfgang Bangerth Aug 24 '20 at 23:00
• Regarding partitioning of the matrix, in the serial version of the code, I store displacement DOFs first and then pressure DOFs, so that I have the 2x2 block matrix format. Partitoning such a matrix across processors would be cumbersome and leads to complex and inefficient communication pattern. I guess I need to change the arrangement of the matrix such the displacement DOFs on each processor are followed immediately by pressure DOFs on the same processor. I would like to know if there are any other efficient ways of implementing this. – Chenna K Aug 24 '20 at 23:01
• @AbdullahAliSivas Right, one can concoct difficult choices of elements where the two variables are discretized in completely different ways (say, a global Fourier basis for $\mathbf u$ and a piecewise polynomial basis for $p$). But I don't think the OP was asking about such cases. – Wolfgang Bangerth Aug 24 '20 at 23:01

You do not like to have two or more different meshes, differently partitioned. That will make massive communication. Degrees of freedom from multiple fields should be as close as possible to each other in adjacency tree, to make interprocessor communication to a minimum.

You have one mesh, but you have DOFs associated with different entities, for example for H1 space, piece-linear continuous, DOFs are on nodes, whereas DOFs for L2 space, price-linear discontinuous, DOFs are on cells. That is for the simple case, for vectorial spaces, like H-div or H-curl, things are a bit more complicated. Of cores, for example, hierarchical space, you can have DOFs on vertices, edges, faces, and cells.

So you partition cells. Sub-entities, i.e. nodes, edges, faces, on the skin of partition are shared. DOFs on shared entities are typically owned by a partition with a lower rank. On other partitions, DOFs on shared entities are so-called ghost DOFs. You can create a special vector with ghost DOFs; you have such vectors in PETSc.

To partition cells, you need to build a graph; then you can use metis, or parameters to partition it. Itself, you have many strategies on how to partition the graph. You can build a graph as well in a different way. You can do it by the numbering of cells and then make an adjacent matrix, by finding neighbour cells through bridge entity. Bridge entity can be node, edge or face. For classical FEM you would use bridge adjacency entity as a vertex. For H-div - L2 formulation bridge adjacency entity should be facing. Since for H-div space, DOFs are on faces (and volumes). When you are using H-curl space, bridge adjacency entity will be an edge. For discontinuous Petrov-Galerkin, bridge adjacency entity will be on a face, since DOFs are on the skeleton.

Moreover, each cell can have weight, if you heterogenous order of approximation. That is needed for load balancing, to distribute work among processors equally.

In the end, there are many solutions, many strategies.

But why to do it by yourself, I can point you to the FEM code, which does it all for you.

• Thank you @likask! The elements (Bezier elements) I am using are not yet available in any FEM package. (Please correct me if I am wrong about this. ) So, I have to do it myself. I use METIS for mesh partitioning in my CFD/FSI code but there I use pressure stabilisation. So, no issues with book-keeping. Please point me to the FEM code concerned with the partitioning and DOFs numbering for mixed elements. Thanks! – Chenna K Aug 24 '20 at 23:14
• You can see parallel mix problem here mofem.eng.gla.ac.uk/mofem/html/mix_transport.html I can point you to other examples which Bernstein-Bezier base, and other coupled problems. For are another case see the problem which three fields and two-element types mofem.eng.gla.ac.uk/mofem/html/cell_forces_8cpp-example.html I've developed code, so I am not fully objective. – likask Aug 25 '20 at 14:12
• Thank you very much for the links! I am going to check these resources right away. – Chenna K Aug 25 '20 at 15:01