10
$\begingroup$

When we have a numerical model that represents a real physical system, and that exhibits chaos (e.g. fluid dynamics models, climate models), how can we know that the model is performing as it should? We cannot compare two sets of model output directly, because even small changes in initial conditions will dramatically change the outputs of individual simulations. We cannot compare the model output directly to observations, either because we can never know with enough detail the initial conditions of the observations, and numerical approximation would anyway cause minor differences that would propagate through the system.

This question is partly inspired by David Ketcheson's question on unit testing scientific code: I'm particularly interested in how regression tests for such models could be implemented. If a minor initial conditions change can lead to major output changes (which may well still be adequate representations of reality), then how can we separate those changes from changes caused by modifying parameters, or implementing new numerical routines?

|cite|improve this question
$\endgroup$
7
$\begingroup$

All you can compare in such cases are the statistics of your solution: averages, higher moments, heat fluxes across the boundary, and other integral quantities. Take a look at one of the many papers discussing turbulence models for the Navier-Stokes equations, for example: they're full to the brim with plots of power spectra, entalpies, entropies, enstrophies, and other words you have never before heard of. All are some integral quantity of the flow and they are compared against the same integral quantities computed from other simulations and/or experiments.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Do you know of a good example paper? Would be a good addition to your answer. $\endgroup$ – naught101 Nov 15 '12 at 4:27
  • $\begingroup$ Not off the top of my head -- I'm not a turbulence modeling person. I would start with some of the more recent papers by Tom Hughes, though, and work from there. $\endgroup$ – Wolfgang Bangerth Nov 15 '12 at 13:35
  • $\begingroup$ I think this is a good example of using summary statistics to determine the regime of a dynamic system: "Statistical inference for noisy nonlinear ecological dynamic systems" $\endgroup$ – Jugurtha Nov 16 '12 at 15:20
4
$\begingroup$

If your code can run in non-chaotic regimes of your underlying problem, especially non-chaotic regimes where you can use the method of manufactured solutions, you should write regression tests that run in these regimes even if they aren't otherwise interesting to you. If these tests fail, then you immediately know that something has gone wrong in your latest code changes. Then you can move on to more physically relevant problems.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I don't think that whole climate models could be run thus way, but perhaps major components could. Something like a super-unit test. Nice idea. $\endgroup$ – naught101 Nov 16 '12 at 20:54
  • 2
    $\begingroup$ But that's the point. Your regression tests should have good code coverage (gcov and the like are your friends) and should run quickly. If you're running an entire climate model as your daily regression test, I suspect you're wasting a lot of time. $\endgroup$ – Bill Barth Nov 16 '12 at 22:24
  • $\begingroup$ I think I was thinking more along the lines of: you run the test initially, and then store a bunch of metrics (as mentioned in Wolfgang's answer). Then you make changes, and run the tests again, and compare the same metrics to the ones you stored last time. If you've improved the model (or model super-component), then theoretically, the metrics should all improve, or at least not worsen dramatically (unless you were over-fitting before, or something, but you can make that decision subjectively). I guess test in this sense are a lot more qualitative, but they could still be very useful. $\endgroup$ – naught101 Sep 3 '14 at 3:57
  • $\begingroup$ As discussed in this answer, I suppose. $\endgroup$ – naught101 Sep 3 '14 at 4:02
2
$\begingroup$

First, I am going to focus on your last sentence, as you touch on a few different things in you question, but I feel it adequately captures what you are asking. If you are changing numerical routines, you shouldn't be changing initial conditions or system parameters until you have validated the new routine from the old one. On the weakest level I see this as comparing some time averaged values over your solution, and them being in agreement (even if the transient behaviors diverged from each other within the chaos). On the strongest level, you would expect the two routines to reproduce the full transient behavior. Which of these you want, and which is acceptable depends on what questions you are asking and what conclusions you are drawing from the solutions.

As far as telling whether a model is "performing as it should", that is an entirely different question. This has nothing to do with the numerical routines you choose. How you build your model, from your simplifying assumptions to your measurements/calculations of parameters, you should be basing all of your decisions on the physics of the problem, and hopefully prior work done on similar cases. You may be able to validate a model with a simple case reproduced in a lab setting, but there are times when even that is non-trivial. If you can't determine an important system parameter to within an order of magnitude, you can't expect anyone to trust the small details you are calculating in the transient local behavior.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.