I need to write a DNS code for simulating reacting flows in turbulent conditions. The code has to be highly scalable, because the computational cost of simulations is expected very large.

My idea is to use C++ and I was wondering if you have suggestions about a very efficient C++ library to manage vector/matrix operations in distributed memory architectures. As an example, I know that PETSc and Trilinos packages are widely used, but I have no idea about the best choice for my case.

Additional details:

  • compressible formulation
  • structured 3D grids
  • finite difference discretization (staggered arrangement)
  • high order FD
  • good scalability on 400-500 cores
  • only for use in my group (no need of complex object oriented design)

Thanks in advance!

  • 2
    $\begingroup$ Do you really need to start from scratch? Are you interested in compressible or incompressible flows? $\endgroup$ Commented Apr 29, 2015 at 22:22
  • 2
    $\begingroup$ What are the scalability and size that you expect? $\endgroup$
    – nicoguaro
    Commented Apr 29, 2015 at 23:27
  • 1
    $\begingroup$ What geometry? What type of numerics? Don't start from scratch unless the implementation choices are the research $\endgroup$ Commented Apr 30, 2015 at 10:32
  • $\begingroup$ The objective is to manage detailed kinetic mechanisms and we need to implement and test several numerical algorithms. Thus, we prefer to have our own code. $\endgroup$
    – alberto
    Commented Apr 30, 2015 at 14:49
  • 1
    $\begingroup$ You cant go wrong with PETSc even if you have to mange grids yourself. However, support for structured grids is pretty mature in PETSc. $\endgroup$
    – stali
    Commented May 5, 2015 at 14:23

2 Answers 2


Like WolfgangBangerth, I strongly recommend that you reconsider your motivation (or your supervisor's motivation) for this goal. First, look at Center for Exascale Simulation of Combustion in Turbulence (ExaCT) web site. They have some proxy applications that may be of use to you in testing your mechanisms, and publications by authors such as John Bell, Joe Oefelein, and Jacqueline Chen can give you an idea of what is the current state of the art in combustion simulation (not just DNS, but also LES, low Mach-number approximations, etc.). Most of these efforts are multi-year, decade-plus efforts. If all you want is something to test some algorithms for detailed chemical kinetics, you might want to either: (a) scale down your ambitions by looking at something simpler, e.g., something like laminar flames or shock tubes or (b) come up with a series of research papers you can write at various intermediate steps in your code development. Either option mitigates your risk. You don't want to spend multiple years developing code and get nothing.

I did my PhD on a related area of combustion research (well, reduced order modeling of detailed kinetic mechanisms). What you're proposing is probably on the order of 2-3 years, at least. For context: a group-mate of mine wrote a laminar finite-difference reacting flame code (I believe it was incompressible), and it took him at least 6 months. Another grad student I knew worked in Ahmed Ghoniem's group and took around a year to write a compressible flow code that was in parallel, but not highly parallelizable. He also wrote the code with a couple other people.

Your first 5 design criteria alone are going to require some serious work, because you're going to have to do something like the following:

  • learn PETSc/Trilinos by hacking one or more examples
  • setup your mesh (probably the easiest part)
  • write out your right-hand side on an incompressible test problem
  • probably write out some approximate Jacobian on a incompressible test problem
  • test for correctness, maybe proof-of-concept scalability
  • add compressibility to right-hand side
  • add compressibility to approximate Jacobian
  • test for correctness, maybe proof-of-concept scalability
  • add reactions to right-hand side
  • add reactions to approximate Jacobian
  • test for correctness on simple reaction case (1-step Arrhenius, 4-step methane, maybe 10-step hydrogen)
  • do a more serious test of scalability
  • start tuning your solver for the laminar compressible case
  • add simple turbulence model to right-hand side
  • add simple turbulence model to approximate Jacobian
  • test for correctness
  • test for scalability
  • add more complicated turbulence model + more testing
  • add more complicated reaction models + even more testing

All of the above description assumes you know exactly what you're doing with the discretizations in space and time, and you know exactly what you're going to use for approximate Jacobians (assuming you need them). I haven't included the inevitable debugging that is going to arise, nor managing the complexity of such a code. If you choose to embark on writing such a code, I urge you to reconsider your remark on object-oriented design. You, as a developer, are going to want to use some sort of organizing principle to manage a large codebase, even if you don't release your software outside of your group. I also haven't included deriving any terms you may not have (e.g., the Jacobian of the reaction terms), coping with performance issues, getting time on clusters, managing the output of your simulations, etc. All of these comments are to say: yes, WolfgangBangerth is right, this endeavor is a multi-year effort.


What you suggest is a multi-year research program. Don't even think about doing this yourself, you will not within several years achieve anything that comes close to the current research. Rather, search the web for codes that are available and adjust them to your needs. A good first place would be the Department of Energy's Computational Combustion research center and network.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.