Our plasma dynamics simulations often produce too much information. During the simulations we record various physical properties on a grid (x,y,z,t) that is as large as (8192x1024x1024x1500), for at least 10 properties. This information is processed after the simulation is completed. With it we

  1. make movies of properties,
  2. perform a Fourier analysis,
  3. calculate average properties.

This simple dumping of as much information as possible worked fine when we studied smaller systems. This gave us the flexibility to interact with the results and decide later what we wanted to do with it. It also allowed us to allot our computational resources (CPU time) to simply running the simulations.

We've begun the process of doing the Fourier analysis on the fly, and filtering for only a select range of length scales. For numerical reasons, we sometimes need to resolve length scales that are smaller than we are actually interested in, so in those cases, this filter helps greatly. We are also exploring various parallel IO libraries, e.g. Parallel I/O options, in particular parallel HDF5.

What strategies are available to maximize the efficiency of data processing?

Is there any benefit to performing all analysis (not including post processing, e.g. movies and plots) on the fly?

I can imagine this issue coming up in other areas of research. For example, you might have a molecular dynamics simulation that needs to evolve for a long time, but you are interested in the brief moment when something interesting is happening. Or in CFD, the early time development might be slow, but once turbulence sets in, you might need a higher time resolution to monitor dynamics.

Are there freely available examples of sophisticated results collection from simulations?

  • $\begingroup$ This might seem like a broad question. If you feel this way, please make suggestions about how I can be more specific. $\endgroup$
    – Yann
    Commented Dec 7, 2011 at 19:00
  • 1
    $\begingroup$ Also look at how some experimental groups deal with this problem. High energy physics (al a CERN) and astrophysics can have even larger scales of data coming in that have to be stored (or even filtered before storing because data comes in faster than it can be written to any storage), distributed, and analyzed. $\endgroup$ Commented Dec 7, 2011 at 20:00

4 Answers 4


I think you might have to split your output to match your targets:

  1. for the movies of properties, you probably don't need the full spatial resolution and all variables. Carefully choose what you want to show and think about the final resolution of the movie you will be showing, it probably won't have 8 billion pixels.
  2. For the Fourier analyses (or things like POD), if they are temporal, you can probably just sample a few hundred points wisely chosen in your domain. If they are spatial, you probably only need a few snapshots and not 1500. And again, not of all properties.
  3. For time averaging, you can just keep adding to the same field and don't have to worry about the time dimension right? Spatial averaging is painful though, especially if you want to look at its evolution over time. But more online processing before dumping the data could reduce the size of it...

This means quite a bit of work to have dedicated outputs instead of a big generic one but this should help keeping the cost and size down. Hope this helps !

Just one more thing I want to add, in general, the full resolution of the data is only needed for restart files, ie files to restart your simulation. You don't need that many of these for a given simulation (let's say 100, so that if something happens between 2 restarts you lose at most 1% of your computation), whereas you probably want to crank up the frequency of output for your movies. And you can do that at just 1/64th of the resolution for example (1 every 4 points in each direction).

  • $\begingroup$ Why is spatial averaging painful? Just do it on-the-fly and write the result, which should be tiny. $\endgroup$ Commented Dec 9, 2011 at 13:00
  • $\begingroup$ @DavidKetcheson Spatial averaging is painful because it requires lots of communication and is potentially influenced by the topology of your domain no? Sure if you have a pure orthogonal grid aligned with your reference frame it is not too bad, but you still have to do some smart combination of computation and MPI_REDUCE because with a grid of that size, you can't just do an ALL_REDUCE on 1 processor I would think... $\endgroup$ Commented Dec 9, 2011 at 16:04
  • 1
    $\begingroup$ Okay, now I understand your comment. But the communication is usually not too bad, since you can average over each process locally and then just reduce a single float per process. In my experience (on a 65K core BlueGene/P), the cost of this is trivial, especially compared to I/O costs. In fact, we do an ALL_REDUCE over the whole 65K cores at every time step and it's very fast. $\endgroup$ Commented Dec 9, 2011 at 18:02
  • $\begingroup$ @DavidKetcheson Actually I now think I misunderstood your point too, and I also was overestimating the cost of the data reduction. What I had in mind was something like a spanwise/azimuthal averaging where you would have to store/output the full 2D data which may or may not be on the same grid as the computational grid. But you are right, the actual cost of the MPI_ALL_REDUCE is not a problem in itself. $\endgroup$ Commented Dec 9, 2011 at 18:45

I think the current masters of this art are the large particle physics experiments (I am most familiar with CDF and D0 because I am old and work at the University of Chicago). They have hardware triggers that discard petabytes (or more) a year. However, this is the whole subject of quantization/discretization, or "throwing away only what you don't need". I am not sure you can give a sensible answer in general. It would be better to narrow down the problem to something like, "I have a PDE simulation discretized in the following way and would like to efficiently downsample".


Peter LePage is pretty famous in lattice-QCD circles for suggesting a method whereby unfeasibly-large lattice grids could be reduced by finding and apply good short ranges analytic solutions.

This is roughly equivalent to noticing that a set of well chosen splines can allow accurate integration with fewer knots than the trapezoid method (except that as in your case you get to take advantage of it over four dimensions at once).

The result is that you trade raw size of the data-set for more computation per node--step, but come out ahead in the end because of the high dimensionality of your problem.

I's not a subject I know well enough to give any decent hints on, but it has worked in some fields in the past.


The question is a bit broad, so I will provide a correspondingly vague answer that suggests possible techniques in such cases.

1) On-the-fly processing, which you are already working on. One way to do on-the-fly processing and yet decouple it from the data-generating step is to generate a cyclic output file that always contains the last N steps, and have the analysis run in a separate process. Obviously you must synchronize the two to prevent a race condition.

2) Choosing the stored data more carefully. This is highly situation-specific, unfortunately.

3) Compress your data before storing it, or use a storage library with integrated compression options, such as HDF5.

4) Store regular checkpoints instead of full output. If you store a full checkpoint every N steps, i.e. enough data to restart the simulation from there, you can reconstruct the missing data in a highly parallel fashion if and when necessary. Note that in the case of Monte-Carlo methods, the checkpoint must include the state of the random number generators. You can actually consider this a highly application-specific compression technique.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.