How to deal with too much data?

Question

Our plasma dynamics simulations often produce too much information. During the simulations we record various physical properties on a grid (x,y,z,t) that is as large as (8192x1024x1024x1500), for at least 10 properties. This information is processed after the simulation is completed. With it we

1. make movies of properties,
2. perform a Fourier analysis,
3. calculate average properties.

This simple dumping of as much information as possible worked fine when we studied smaller systems. This gave us the flexibility to interact with the results and decide later what we wanted to do with it. It also allowed us to allot our computational resources (CPU time) to simply running the simulations.

We've begun the process of doing the Fourier analysis on the fly, and filtering for only a select range of length scales. For numerical reasons, we sometimes need to resolve length scales that are smaller than we are actually interested in, so in those cases, this filter helps greatly. We are also exploring various parallel IO libraries, e.g. Parallel I/O options, in particular parallel HDF5.

What strategies are available to maximize the efficiency of data processing?

Is there any benefit to performing all analysis (not including post processing, e.g. movies and plots) on the fly?

I can imagine this issue coming up in other areas of research. For example, you might have a molecular dynamics simulation that needs to evolve for a long time, but you are interested in the brief moment when something interesting is happening. Or in CFD, the early time development might be slow, but once turbulence sets in, you might need a higher time resolution to monitor dynamics.

Are there freely available examples of sophisticated results collection from simulations?

Answer 1

I think you might have to split your output to match your targets:

1. for the movies of properties, you probably don't need the full spatial resolution and all variables. Carefully choose what you want to show and think about the final resolution of the movie you will be showing, it probably won't have 8 billion pixels.
2. For the Fourier analyses (or things like POD), if they are temporal, you can probably just sample a few hundred points wisely chosen in your domain. If they are spatial, you probably only need a few snapshots and not 1500. And again, not of all properties.
3. For time averaging, you can just keep adding to the same field and don't have to worry about the time dimension right? Spatial averaging is painful though, especially if you want to look at its evolution over time. But more online processing before dumping the data could reduce the size of it...

This means quite a bit of work to have dedicated outputs instead of a big generic one but this should help keeping the cost and size down. Hope this helps !

Just one more thing I want to add, in general, the full resolution of the data is only needed for restart files, ie files to restart your simulation. You don't need that many of these for a given simulation (let's say 100, so that if something happens between 2 restarts you lose at most 1% of your computation), whereas you probably want to crank up the frequency of output for your movies. And you can do that at just 1/64th of the resolution for example (1 every 4 points in each direction).

Answer 2

I think the current masters of this art are the large particle physics experiments (I am most familiar with CDF and D0 because I am old and work at the University of Chicago). They have hardware triggers that discard petabytes (or more) a year. However, this is the whole subject of quantization/discretization, or "throwing away only what you don't need". I am not sure you can give a sensible answer in general. It would be better to narrow down the problem to something like, "I have a PDE simulation discretized in the following way and would like to efficiently downsample".

Answer 3

Peter LePage is pretty famous in lattice-QCD circles for suggesting a method whereby unfeasibly-large lattice grids could be reduced by finding and apply good short ranges analytic solutions.

This is roughly equivalent to noticing that a set of well chosen splines can allow accurate integration with fewer knots than the trapezoid method (except that as in your case you get to take advantage of it over four dimensions at once).

The result is that you trade raw size of the data-set for more computation per node--step, but come out ahead in the end because of the high dimensionality of your problem.

I's not a subject I know well enough to give any decent hints on, but it has worked in some fields in the past.

Answer 4

The question is a bit broad, so I will provide a correspondingly vague answer that suggests possible techniques in such cases.

1) On-the-fly processing, which you are already working on. One way to do on-the-fly processing and yet decouple it from the data-generating step is to generate a cyclic output file that always contains the last N steps, and have the analysis run in a separate process. Obviously you must synchronize the two to prevent a race condition.

2) Choosing the stored data more carefully. This is highly situation-specific, unfortunately.

3) Compress your data before storing it, or use a storage library with integrated compression options, such as HDF5.

4) Store regular checkpoints instead of full output. If you store a full checkpoint every N steps, i.e. enough data to restart the simulation from there, you can reconstruct the missing data in a highly parallel fashion if and when necessary. Note that in the case of Monte-Carlo methods, the checkpoint must include the state of the random number generators. You can actually consider this a highly application-specific compression technique.