I am working on some medium to large scale finite element codes. By using established and available tools I am able to have an algorithm that scales well up to about 10,000 cores. Investigating scaling beyond this requires investigating larger meshes, which leads to my problem.

Once meshes start getting large (in range of 100s of gigabytes to terabyte ranges), simply getting them to a cluster environment can completely overwhelm any cost of solving the resulting system. 100GB-1TB mesh sizes aren't especially large sizes by today's standard of nodes as well, which can have upwards of 64GB of memory each (and in many cases more than that)

SO how is this commonly handled? Are there common ways to improve the bandwidth in transferring data to a cluster? Do you just need to be on an incredibly high bandwidth connection, or physically ship a drive containing all the data you want?

As a followup question: if I could re-engineer this, would it solve the problem to rely more on automatic mesh refinement so that we start with a smaller starting mesh and refine at need in-memory?

  • $\begingroup$ Might be better to start with a small mesh, partition/distribute and then refine. BTW what mesh generation software are you using for 100GB meshes? $\endgroup$ – stali Feb 27 '17 at 17:38
  • $\begingroup$ I've pushed gmsh past that limit before by using its "refine" capability: gmsh -refine input_mesh.msh -o output_mesh.msh of course this would have to fit in memory of a single node, so I used some very large memory nodes to get very large meshes like this. Going beyond that with more realistic meshes is not something I've tried. $\endgroup$ – Reid.Atcheson Feb 27 '17 at 17:47

My answer is primarily opinion-based, given my experience. In my work, I haven't (yet) dealt with meshes quite as large as what you're describing. However, I've seen large enough meshes to hint that your problems might be my problems in the future, so I've pondered the problem.

Here are some comments/suggestions ordered from obvious/easy-to-implement to more involved.

  • Make sure you're using binary mesh files, not ASCII. Depending on exact implementation, ASCII files can be significantly larger and also take significantly more time to read, due to the cost of string parsing.

  • Parallel mesh file formats such as Parallel NetCDF can help spread storage and I/O of the mesh on separate compute nodes. Of course, that doesn't help you get the data to the cluster.

  • You could transfer a smaller initial mesh and do some command line/script-based refinement on the remote cluster. Of course, you still need to generate a single huge mesh on the cluster, whether that mesh is stored on disk or in memory, as I know of no parallel mesh refinement tools. (Edit: @VorKir brought my attention to mfem, which appears to include a very nice parallel meshing library made by the good people at Lawrence Livermore National Laboratory.)

  • In my opinion, Domain Decomposition Methods present a critical mathematical framework for implementing massively parallel finite element methods. There's more than one way to skin this cat, since DDMs give you the flexibility to do a lot of different things. However, one of the most important features of DDMs is that they can be formulated to allow non-matching meshes at interfaces between domains. In the context of mesh refinement, this allows the mesh for each domain to refined independently of all other domains, avoiding the need for a truly parallel mesher. The final solution could look something like:

    1. Generate coarse mesh from original geometry.

    2. Partition coarse mesh into domains and distribute to compute nodes.

    3. Perform mesh refinement independently on each domain.

    4. Run simulation using DDM-enabled finite element method.

  • $\begingroup$ There are parallel mesh refinement tools. For example, MFEM from mfem.org has this feature. So, DDM is not the only option, it is powerful but sometimes complicated. $\endgroup$ – VorKir Feb 27 '17 at 22:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.