Storing Raw Simulation Data or Truncated Data?

Question

I have a simulation that can generate quite a bit of data when it runs, for example $650\cdot 400 \cdot 400$ floating point numbers. Without compression, that's a few gigabytes worth if I want to save the whole simulation and not just some data extracted from it. However, with some methods, like Crank-Nicolson, we know the maximum error that can occur, (it has second order accuracy). Suppose I calculate it, then round or truncate all the values in my $650 \cdot 400 \cdot 400$ matrix to the same number of digits (or the same number of digits plus 1): $$tolerence = .00004$$ $$example = .45435354354 \rightarrow example_{rounded} = .454354$$

If I know a maximum value I can now used fixed point math and potentially get some very good compression using algorithms for integers (as storing compressed floating points can be difficult, and fixed-point numbers work similarly to integers "under the hood"). This would also allow me to use binary storage without much difficulty. In any way is this inadvisable? Might I loose important information, even if it is below tolerance? Will this affect the analysis I do on the data (even if tolerance [error] is preserved [is it?])?

Answer 1

I would recommend keeping the computation in your original precision and reducing the accuracy just when you write, unless the computation is too slow. If you reduce the precision of the computation, you may need to worry about the accuracy of the linear solve.

First, you should consider whether you need to write all of the time steps. I'd guess that Crank-Nicolson needs a smaller time step than your analysis/visualization. Casting the data to a lower-accuracy floating- or fixed-point when you write it to file is similar. Both of these are used by major scientific codes (e.g., NekRS).

If you need more compression, I'd recommend a floating-point compressor meant for scientific data. This is a hot topic since some of the big simulations can generate petabytes per run. Off the top of my head, there's ZFP and SZ. Both are available as libraries, so they're probably even easier to use than fixed-point.

I doubt using integer compression on fixed-point numbers will be effective. Most integer compression schemes assume the data has particular structure. For example, that the data only contains a few distinct values or that most of the values are small. Neither fixed-point and floating-point satisfy these types of assumptions.