Use of machine learning in computational fluid dynamics

Question

Background:
I have only built one working numeric solution to 2d Navier-Stokes, for a course. It was a solution for lid-driven cavity flow. The course, however, discussed a number of schemas for spatial discretizations and time discretizations. I have also taken more symbol-manipulation coursework applied to NS.

Some of the numeric approaches to handle conversion of the analytic/symbolic equation from PDE to finite difference include:

• Euler FTFS, FTCS, BTCS
• Lax
• Midpoint Leapfrog
• Lax-Wendroff
• MacCormack
• offset grid (spatial diffusion allows information to spread)
• TVD

To me, at the time, these seemed like "insert-name finds a scheme and it happens to work". Many of these were from before the time of "plentiful silicon". They are all approximations. In the limit they. in theory, lead to the PDE's.

While Direct Numerical Simulation (DNS) is fun, and Reynolds Averaged Navier-Stokes (RANS) is also fun, they are the two "endpoints" of the continuum between computationally tractable, and fully representing the phenomena. There are multiple families of approaches that live interior to these.

I have had CFD professors say, in lecture, that most CFD solvers make pretty pictures, but for the most part, those pictures do not represent reality and that it can be very tough, and take lots of work, to get a solver solution that does represent reality.

The sequence of development (as I understand it, not exhaustive) is:

1. start with the governing equations -> PDE's

2. determine your spatial and temporal discretization -> grid and FD rules

3. apply to the domain including initial conditions and boundary conditions

4. solve (lots of variations on matrix inversion)

5. perform gross reality checks, fit to known solutions, etc..

6. build some simpler physical models derived from analytic results

7. test them, analyze, and evaluate

8. iterate (jumping back to either step 6, 3, or 2)

Thoughts:
I have recently been working with CART models, oblique trees, random forests, and gradient boosted trees. They follow more mathematically derived rules, and the math drives the shape of the tree. They work to make discretized forms well.

Although these human-created numeric approaches work somewhat, there is extensive "voodoo" needed to connect their results to the physical phenomena they are meant to model. Often the simulation does not substantially replace real-world testing and verification. It is easy to use the wrong parameter, or not account for variation in geometry or application parameters experienced in the real world.

Questions:

• Has there been any approach to let the nature of the problem define
  the appropriate discretization, spatial and temporal differencing scheme, initial conditions, or solution?
• Can a high definition solution coupled with the techniques of machine learning be used to make a differencing scheme that has much larger step sizes but retains convergence, accuracy, and such?
• All of these schemes are accessibly "humanly tractable to derive" - they have a handful of elements. Is there a differencing scheme with thousands of elements that does a better job? How is it derived?

Note: I will follow up with the empirically intialized and empirically derived (as opposed to analytically) in a separate question.

UPDATE:

1. Use of deep learning to accelerate lattice Boltzmann flows. Gave ~9x speedup for their particular case

   Hennigh, O. (in press). Lat-Net: Compressed Lattice Boltzmann Flow Simulations using Deep Neural Networks. Retrieved from: https://arxiv.org/pdf/1705.09036.pdf

   Repo with code (I think):
   https://github.com/loliverhennigh/Phy-Net

2. About 2 orders of magnitude faster than GPU, 4 orders of magnitude, or ~O(10,000x) faster than CPU, and same hardware.

   Guo, X., Li, W. & Ioiro, F. Convolutional Neural Networks for Steady Flow Approximation. Retrieved from: https://autodeskresearch.com/publications/convolutional-neural-networks-steady-flow-approximation

3. Others who have looked into the topic about 20 years ago:

   Muller, S., Milano, M. & Koumoutsakos P. Application of machine learning algorithms to flow modeling and optimization. Center for Turbulence Research Annual Research Briefs 1999 Retrieved from: https://web.stanford.edu/group/ctr/ResBriefs99/petros.pdf

Update (2017):
This characterises the use of non-gradient methods in deep learning, an arena which has been exclusively gradient based. While the direct implication of activity is in deep learning, it also suggests that GA can be used as an equivalent in solving a very hard, very deep, very complex problem at the level consistent with or superior to gradient descent based methods.

Within the scope of this question, it might suggest that a larger-scale, machine-learning based attack might allow "templates" in time and space that substantially accelerate convergence of gradient-domain methods. The article goes as far as to say that sometimes going in the direction of gradient descent moves away from the solution. While in any problem with local optima or pathological trajectories (most high-value real-world problems have some of these) it is expected that the gradient isn't globally informative, it is still nice to have it quantified and validated empirically as it was in this paper and the ability to "jump the bound" without requiring "reduction of learning" as you get in momentum or under-relaxation.

Update (2019):
It seems that google now has a contribution "how to find a better solver" piece of the AI puzzle. link This is a part of making the AI make the solver.

Update (2020b):
And now they are doing it, and doing it well...
https://arxiv.org/pdf/1911.08655.pdf

Is that a 100x speedup for Navier-Stokes solution??

It could be argued that they could then deconstruct their NN to determine the actual discretization. I particularly like figure 4.

Update 2020 link

Update 2022 - Physics Informed neural networks allow state compression by a factor of 85x with only slight loss. (link) One of the values of being able to stop, store the state, move it to another computer, and be able to pick it up and restart at the stop-point and not the initial condition is that it can allow reproducibility. A decent near-estimate of a state means the compute engine can get to the actual with only a little work. A ML-based "picture" that is good enough can act just like all the compute overhead from the initial state to the good enough estimate.

Update 2022:
ML systems are now becoming much better at pushing back the pareto frontier on chaotic systems. The demo is a flame front. This suggests there is a form implicit to the learner that captures and predicts the chaotic part in a way that analytical approaches do not.

https://www.quantamagazine.org/machine-learnings-amazing-ability-to-predict-chaos-20180418/
https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.120.024102

Answer 1

It's a long-running joke that CFD stands for "colorful fluid dynamics". Nevertheless, it is used -- and useful -- in a wide range of applications. I believe your discontent stems from not sufficiently distinguishing between two interconnected but different steps: creating a mathematical model of a physical process and solving it numerically. Let me comment on these a bit:

1. No mathematical (or really, any) model of physical reality is ever correct; at best it's useful for predicting the results of measurements in a very precisely demarcated (but hopefully large) set of circumstances. This includes the fact that it must be possible to obtain such a prediction given a specific configuration; this is why we have a whole hierarchy of models from quantum field theory to Newtonian mechanics. In particular, the Navier-Stokes equations do not describe fluid flow, they give a prediction of specific aspects of the behavior of certain fluids under certain conditions.

2. For the more complicated mathematical models (such as the Navier-Stokes equations), you can never obtain an exact solution (and hence prediction), but only a numerical approximation. This is not such a bad thing as it sounds, since the measurements you want to compare them with are themselves never exact. Just as in the choice of models, there's a trade-off between accuracy and tractability -- it makes no sense to spend time or money on getting a more accurate solution than needed. At this point, it becomes purely a question on how to approximate numerically the solution of (in this case) a partial differential equation, which is the subject of a whole mathematical field: numerical analysis. This field is concerned with proving error estimates for certain numerical methods (again, under certain, explicitly specified, conditions). Your statement "insert-name finds a scheme and it happens to work", is grossly unfair -- it should be "insert-name finds a scheme and proves that it works". Also, these schemes are not pulled out of thin air -- they are derived from well-understood mathematical principles.

   (For example, finite difference schemes can be derived using Taylor-approximations of a given order. It is certainly possible to -- and some people do -- obtain very high-order difference schemes and implement them, but there's a law of diminishing returns: this can only be automated partially, and hence takes a lot of effort, and certain increasingly restrictive conditions must be satisfied to actually get the corresponding higher accuracy out of them. Also, at some point it's better to use a different scheme altogether such as spectral methods.)

The common theme here is that both models and numerical schemes come with a range of applicability, and it is important to pick the right combination for a given purpose. This is precisely why a computational scientist needs to know both the domain science (to know which model is valid in which situation) and mathematics (to know which method is applicable to which model, and to which accuracy)! Ignoring these "use only as directed" labels leads to producing the kind of "computational bullshit" (in the technical sense of Harry Frankfurt) your CFD professors referred to.

As to why use a computational model when you have a physical model (such as a wind tunnel): One reason is that running software can be orders of magnitude cheaper than creating a model and putting it in a wind tunnel. Also, it's usually not an either-or: For example, when designing a car or an airplane, you would run hundreds or thousands of simulations to narrow things down, and then only for the final candidate(s) put a model into a wind tunnel.

---

Update:

Using machine learning instead of numerical simulation is like saying "having no model is better than having an approximate model", which I doubt anyone in fluid dynamics (or any other field) would agree with. That being said, it's certainly possible (and actually done) to use machine learning to select unknown "geometry or application parameters" based on agreement with measured data; however, here as well model-based methods such as uncertainty quantification or (Bayesian) inverse problems usually perform much better (and are based on rigorous mathematical principles). Selecting numerical parameters such as step size or order of the method using machine learning is also possible in principle, but I fail to see the benefit since there's a mathematical theory that tells you precisely how to pick these parameters based on your (mathematical) model.

Update 2:

The paper you link to is about computer graphics, not computational science: their goal is not to have an accurate simulation (i.e., a numerical solution of a mathematical model) of a physical process, but something that just looks like one to the naked eye (an extreme case of "colorful fluid dynamics"...) -- that is a very different matter. In particular, there is no error bound for the output of the trained network compared to the corresponding solution to the Navier-Stokes equations, which is an indispensable part of any numerical method.

---

(And your first question starts from a false premise: in every approach, the problem determines the model, the model determines the discretization, the discretization determines the solver.)

Answer 2

I think you are mixing a couple different ideas that are causing confusion. Yes, there are a wide variety of ways to discretize a given problem. Choosing an appropriate way may look like "voodoo" when you are learning these things in class, but when researchers choose them, they do so drawing on the combined experience of the field, as published in literature. Therefore they make much more informed choices than a student could.

Question 1: If you are solving a problem, and you switch from one scheme to another, your run time will change, the convergence criteria may change, or your asymptotic behavior, but a very important point is that your final converged solution should NOT change. If it does, you either need to refine your meshes, or there is something wrong with your numerical scheme. Perhaps you could use some optimization algorithm to create your numerical schemes and improve performance for a specific class of problems, but many times the hand derived schemes are created with mathematically provable optimal convergence/asymptotic behavior for the number of terms involved or mesh type used.

Now the above paragraph doesn't account for things like different turbulence models, which are different mathematical formulations/approximations of the physics, so are expected to have different solutions. These are again highly studied in the literature, and I don't think programs are at the point that they can look at physical phenomena and produce a mathematical model that properly predicts the response of similar physical systems.

Question 2: Yes, you could derive a scheme that uses the entire mesh at once, using some computer code to do it. I even feel safe saying that for some meshes, such codes exist, and could give you your scheme in a matter of a couple hours (once you find the code that is). The problem is that you will never beat Nyquist. There is a limit to how large of time steps you take, depending on the max frequency of the response of your system, and a limit to how large of mesh cells/elements you can have depending on the spatial frequencies of the solution.

That doesn't even account for the fact that often the computational work involved in using a more complex scheme is often non-linear with complexity. The reason most students learn RK4 methods for time integration is that when you start going to methods with a higher order than that, you gaining more evaluations of your derivative faster than you gain orders of your method. In the spatial realm, higher order methods greatly increase matrix fill-in, so you need less mesh points, but the work you do inverting the sparse matrix increases greatly, at least partially offsetting your gains.

I am not sure what you are referring to in question three. Are you talking about turning close solutions to a problem into better solutions? If so I recommend some light reading on multigrid. If you are asking about turning decent numerical schemes into amazing ones, I think the rest of my answer at least touches on that.