Working with large mesh files

Question

I am working on some medium to large scale finite element codes. By using established and available tools I am able to have an algorithm that scales well up to about 10,000 cores. Investigating scaling beyond this requires investigating larger meshes, which leads to my problem.

Once meshes start getting large (in range of 100s of gigabytes to terabyte ranges), simply getting them to a cluster environment can completely overwhelm any cost of solving the resulting system. 100GB-1TB mesh sizes aren't especially large sizes by today's standard of nodes as well, which can have upwards of 64GB of memory each (and in many cases more than that)

SO how is this commonly handled? Are there common ways to improve the bandwidth in transferring data to a cluster? Do you just need to be on an incredibly high bandwidth connection, or physically ship a drive containing all the data you want?

As a followup question: if I could re-engineer this, would it solve the problem to rely more on automatic mesh refinement so that we start with a smaller starting mesh and refine at need in-memory?

Answer 1

My answer is primarily opinion-based, given my experience. In my work, I haven't (yet) dealt with meshes quite as large as what you're describing. However, I've seen large enough meshes to hint that your problems might be my problems in the future, so I've pondered the problem.

Here are some comments/suggestions ordered from obvious/easy-to-implement to more involved.

• Make sure you're using binary mesh files, not ASCII. Depending on exact implementation, ASCII files can be significantly larger and also take significantly more time to read, due to the cost of string parsing.

• Parallel mesh file formats such as Parallel NetCDF can help spread storage and I/O of the mesh on separate compute nodes. Of course, that doesn't help you get the data to the cluster.

• You could transfer a smaller initial mesh and do some command line/script-based refinement on the remote cluster. Of course, you still need to generate a single huge mesh on the cluster, whether that mesh is stored on disk or in memory, as I know of no parallel mesh refinement tools. (Edit: @VorKir brought my attention to mfem, which appears to include a very nice parallel meshing library made by the good people at Lawrence Livermore National Laboratory.)

• In my opinion, Domain Decomposition Methods present a critical mathematical framework for implementing massively parallel finite element methods. There's more than one way to skin this cat, since DDMs give you the flexibility to do a lot of different things. However, one of the most important features of DDMs is that they can be formulated to allow non-matching meshes at interfaces between domains. In the context of mesh refinement, this allows the mesh for each domain to refined independently of all other domains, avoiding the need for a truly parallel mesher. The final solution could look something like:

  1. Generate coarse mesh from original geometry.

  2. Partition coarse mesh into domains and distribute to compute nodes.

  3. Perform mesh refinement independently on each domain.

  4. Run simulation using DDM-enabled finite element method.