2
$\begingroup$

Suppose we want to solve an FEM problem in terms of HPC. What is the most usual way to do it:

  1. Using an open-source software like mfem,deal.ii etc.. or,

  2. Assembly the system by your own(read mesh file,create stiffness matrix etc..) and use a high performance software like hypre,PETSc to solve it.

Improve this question
$\endgroup$

5 Answers 5

Reset to default
5
$\begingroup$

The correct answer to your question, IMHO, is "depends on your target and your problem at hand".

1.) If your target is to simulate a large-scale problem on HPC and if you know of an existing code which can model the physics of your problem readily, then use the existing code.

2.) If an existing code does not yet support the physics of your problem but offers a lot of utilities, then build the required features on top of that library.

3.) If you are a researcher working on Computational Physics or Computational Engineering, then having your own code might be a good idea. All the current opensource libraries have some or other limitations: they may not support some element types or material laws, or their way of application of BCs is not robust.

Still, you should make use of the libraries such as PETSc, Eigen, Boost and VTK, and develop the code for solving the physics of the problem using a numerical scheme of your choice.

Note that even with all the third-party libraries, it takes a lot of effort to develop the code on your own. But it is necessary if you are working on new numerical schemes.

Improve this answer
$\endgroup$
1
  • $\begingroup$ I completely agree with you. $\endgroup$
    – ConvexHull
    Commented Aug 3, 2020 at 16:53
4
$\begingroup$

Why would you want to do things on your own? The libraries you mention have all been run on 10,000+ cores and under the hood use PETSc, Trilinos, hypre, ... for the solution of linear systems or use matrix-free approaches. You would have to invest tens of man-years of work to implement the functionality and optimizations that has gone into these libraries -- you won't be able to compete, and nor should you: all you will achieve is a software that does what others have already done for years, and will do so slower.

(Disclaimer: I'm one of the principal authors of deal.II.)

Improve this answer
$\endgroup$
5
  • $\begingroup$ I see your point,however if someone read the mesh file with information about element connectivity and nodes coordinates it is easy to assembly the FEM equation Ku=b. METIS/PaMetis,Hypre or PETSc can handle the parellazation part. That why I'm asking $\endgroup$
    – spyros
    Commented Aug 2, 2020 at 10:26
  • $\begingroup$ One possible reason is that these libraries require C++ which not all of us are familiar with. Is it possible to use these libraries coming from a C or Fortran background? $\endgroup$
    – vibe
    Commented Aug 3, 2020 at 4:52
  • $\begingroup$ @spyros Sure, it's easy to implement a code yourself that uses linear elements on triangles for the Laplace equation. But what if instead you want to solve a more complex, coupled set of equations? But what if you want to use higher order elements, say for the Stokes equation? What if you want to use curved elements to better approximate the boundary? What if you decide use a matrix-free solver? What if you want to use GPUs? Each of these can be done with a few or a few dozen lines of code in libraries, but will take you months to implement from scratch. $\endgroup$
    – Wolfgang Bangerth
    Commented Aug 3, 2020 at 12:58
  • $\begingroup$ @spyros In other words, all you will ever be able to solve is the Laplace equation on linear elements, while everyone else is solving real problems. $\endgroup$
    – Wolfgang Bangerth
    Commented Aug 3, 2020 at 12:58
  • $\begingroup$ @vibe What you're saying is that you're willing to invest months or years of your life into implementing something that others have already implemented with more features than you can probably ever do. And that you'd rather do that instead of spending a few weeks learning a new programming language -- a skill set that might be useful to you for many years to come. That's like saying that you'd rather plow your field with your oxen than harvest a field that someone else has already plowed for you, because you don't want to learn how to drive a tractor :-) $\endgroup$
    – Wolfgang Bangerth
    Commented Aug 3, 2020 at 13:02
1
$\begingroup$

I would highly suggest you go with an available FEM open-source library (say deal.II, FENICS, MFEM, etc.) instead of writing your own FEM code and then using PETSC as the underlying parallel algebra library. First, the majority of open source HPC FEM code already use either PETSC or Trilinos under the hood (deal.II supports both, FENICS uses PETSC, etc.). Additionnaly, I think you underestimate the complexity of writing your own parallel FEM code. It might doable in a few months for a simple Poisson equation if you already know what you are doing, but if you want to solve non-linear vector-equations and support more than just P1 elements, it will take you considerable time to write and test a piece of software that can reach the same degree of performance as the majority of open source libraries. Additionnaly, this re-inventing of the wheel leads to results that are very difficult to publish because they bring nothing new. You have to think about the amount of elements you will need to re-program : parameter parsing, mesh parsing, result output in parallel (which is not a trivial task), etc.

I would really advise agaisnt writing your own FEM platform from scratch. It is better to contribute to an existing platform to add the functionnalities that you are after. Although you need to invest the time to learn a library, you generally recuperate this initial investment as soon as you do something more complex.

Improve this answer
$\endgroup$
0
$\begingroup$

You will have more control over things if you use PETSc. The most difficult part of writing a performant FE code is parallel assembly and solve and PETSc takes care of both. PETSc even has routines for managing unstructured meshes (DMPLEX).

With other codes your choice of programming language, type of meshes/elements, etc. are somewhat limited. PETSc also has the backing of DOE and mostly likely will be around even 20-30 years from now. Same may not be true for other projects. Right now there are too many of them and I doubt all will be actively maintained after 10-15 years.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ MFEM is also backed by DOE. deal.II is backed by the NSF and half a dozen other funding agencies both in the US and abroad. deal.II also has been around for 23 years now and has a larger developer and user community than it ever had, so the argument that "it might go away" really doesn't make much sense. All of the big FEM libraries are so large now that they will be around for the indefinite future -- certainly for longer than any one research project thinking about using them. $\endgroup$
    – Wolfgang Bangerth
    Commented Aug 2, 2020 at 0:22
0
$\begingroup$

To say you should just use open source is quite naive. I think it depends on what you are interested in.

If you are interested in code development which should be published later I highly recommend implementing your own stuff.

Here are some arguments:

  1. You know what you are doing!
  2. It often appears that something is not exactly implemented as you thought it would be.

If you are interested in the applications, where you only want to make changes to physical modelling or using different equation systems then simply use open source libraries.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I've made that point many times in my career: If you think that you can write code that is more correct than the big libraries (your point 2), you're certainly mistaken. The big libraries all runs thousands of tests on each commit. These tests themselves equate to many years of work. $\endgroup$
    – Wolfgang Bangerth
    Commented Aug 3, 2020 at 13:06
  • 1
    $\begingroup$ Dear @WolfgangBangerth with this point i don't claim, that something in the library is less correct than your own implementations. I only want to highlight that quite often algorithms indeed are not exactly doing what is documented. From my experiences, there is not always one correct way to implement something. This might often result in small or neglectible differences. However, some algorithms may respond highly sensitiv on such differences. Think for example of a bad conditioned problem, where a different sequence of function calls resulting in quite different results. $\endgroup$
    – ConvexHull
    Commented Aug 3, 2020 at 15:40
  • 1
    $\begingroup$ Moreover, would you claim that only using "blackbox" funtions caused you to have such a deep understanding of numerics today? I think it is quite important for a good computional scientist to implement algorithms "at least once". $\endgroup$
    – ConvexHull
    Commented Aug 3, 2020 at 16:33
  • $\begingroup$ Yes, you implement a toy problem once, and then you use someone else's "real" implementation :-) $\endgroup$
    – Wolfgang Bangerth
    Commented Aug 4, 2020 at 18:31
  • $\begingroup$ As for small differences: Yes, different implementations will have minor differences in output (at least for floating point and iterative algorithms). But what consequence does that have? If you're relying on a specific output within round-off or iteration error, you're probably doing something wrong. And if you think that it's easier to just re-implement everything that trying to understand what others are doing, then that's probably also not productive -- if there's a bug in someone's implementation, it's faster to fix it than to re-implement the whole thing. $\endgroup$
    – Wolfgang Bangerth
    Commented Aug 4, 2020 at 18:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.