Compressing floating point data

Question

Are there any tools specifically designed for compressing floating point scientific data?

If a function is smooth, there's obviously a lot of correlation between the numbers representing that function, so the data should compress well. Zipping/gzipping binary floating point data doesn't compress it that well though. I am wondering if there are methods specifically developed for compressing floating point data.

Requirements:

• Either lossless compression or the possibility to specify a minimum number of digits to retain (for some applications double might be more than what we need while float might not have enough precision).

• Well tested working tool (i.e. not just a paper describing a theoretical method).

• Suitable for compressing 1D numerical data (such as a time series)

• Cross platform (must work on Windows)

• It must be fast---preferably not much slower than gzip. I found that if I have the numbers stored as ASCII, gzipping the file can speed up reading and processing it (as the operation might be I/O bound).

I'd especially like to hear from people who have actually used such a tool.

Answer 1

Try out Blosc. It is in many cases faster than memcopy. Think about that for a second. . . wicked.

It is super stable, highly-vetted, cross-platform, and performs like a champ.

Answer 2

I got good results using HDF5 and its GZIP filter.

The HDF5 also provides an SZIP filter which achieves better results for some scientifica data-sets.

In my experience the choice of compressions depends heavily on the kind of data and benchmarking is probably the only way to make a good choice.

BTW, third-party filters for HDF5 include BLOSC, BZIP2, LZO, LZF, MAFISC.

Answer 3

Arguably, you can interpret regression or transform methods (Fourier transform, Chebyshev transform) methods as "compression" for time-series or 1D function data. Remez's algorithm would be another candidate. In that case, using something like regression, FFT, or Chebyshev via FFT would work for your purposes. That said, none of these methods works on time series data with arbitrary structure. For example, with FFT, you assume periodicity, and any sort of discontinuities in the data (or lack of periodicity) will lead to Gibbs' phenomenon. Similarly, with Chebyshev transforms, the assumption is that the data describes a function on $[-1,1]$.

Depending on the underlying function, you may be able to fit the data to a functional form without error, requiring fewer coefficients to describe the functional form than you have data point (leading to the compression). Error results exist for some of these methods, although I don't know if any of them will give you a priori (or a posteriori) bounds or estimates on the error.

You could also look at methods developed specifically for the compression of floating point numbers, like FPC and related algorithms. See the papers here, here, here, here, and here, along with a web page containing old source code here.

Answer 4

HDF5 can use a "shuffling" algorithm where the bytes for N floating point numbers are rearranged so that the first bytes of the N numbers come first, then the 2nd, and so on. This produces better compression ratios after gzip is applied, as it is more likely to produce longer sequences of the same value. See here for some benchmarks.

Answer 5

We have been using ZFP with HDF5 for our medical imaging data. It is made for lossy, floating point compression.

We're running it on literally everything, and have more than 40TB of data stored (and being used!). It is fast enough to save our data real-time, and we can specify the required precision, so while the format is lossy, we're not seeing any differences in our final outputs.

Answer 6

SZ (developed by Argonne in 2016) could be a good choice.

SZ: Fast Error-Bounded Floating-point Data Compressor for Scientific Applications https://collab.cels.anl.gov/display/ESR/SZ

Answer 7

Possible methods, that can be used for floating-point compression:

• Transpose 4xN for float and 8xN for double + lz77
  Implementation: Floating point compression in TurboTranspose
  see also error-bounded lossy compression

• Predictor (ex. Finite Context Method) + encoding (ex. "integer compression").
  Implementation: Floating point compression in TurboPFor
  including special compression for time series.

• when possible, convert all floating point numbers to integers (ex. 1.63 -> 163), then use integer compression

• You can test all these methods with your data using the icapp tool for linux and windows.

Answer 8

If a function is smooth, there's obviously a lot of correlation between the numbers representing that function, so the data should compress well.

Perhaps the format you require need to store just the offsets from value to neighboring value.

Alternately, maybe you could make use of the frequency domain, perhaps even saving these values as a lossless audio file such as "flac lossless", as you require some of the same properties to a sound.

However, I'm going to take a different approach to attempting to answer the question which I hope can be of some help. As what you are saying is also that the minimum description length to represent this data is less than providing all the data points.

https://en.wikipedia.org/wiki/Minimum_description_length

Effectively a program, computer code, is a good example. And if you don't mind that somthing thats primarily data working by executing, and so also being code, then you could compress your floating point values into somthing like a function or formuli.

Doing this particularly well automatically, and in a realistic amount of compute, is beyond hard. However the Wolfram Language provides some functionality to attempt this:

https://reference.wolfram.com/language/ref/FindSequenceFunction.html https://reference.wolfram.com/language/ref/FindGeneratingFunction.html https://reference.wolfram.com/language/ref/FindFormula.html

https://reference.wolfram.com/language/ref/RSolve.html

Answer 9

Why not just save float32 / float16 ? In numpy,

A.astype( np.float32 )  # 100M: 200 msec imac
A.astype( np.float16 )  # 100M: 700 msec

These won't do if you're simulating the Butterfly effect in chaos theory, but they're understandable, portable, "do not require any work on my part". And compression 2:1 / 4:1 over float64 is hard to beat :)

Notes:

"Array type float16 is unsupported in np.linalg"; you'll have to expand it to 32 or 64 after reading it in.

To see how floating-point parameters differ,

import numpy as np
for f in [np.float64, np.float32, np.float16]:
    print np.finfo(f)

For a plot of a trivial test case comparing float 64 32 and 16, see here .

Answer 10

This is one of the first results that come up when you search in google. My use case is whole slide images (i.e. huge TIFF files with JPEG compression) over which I had to perform some post-processing and save the results. So I have huge images (e.g. 100000 x 90000 pixels), 2 channels w/. float16 information.

For my specific application it made sense to divide the image in patches, so I tiled the image in 512 x 512 pixel patches. In this setting, three compression methods stood out:

• LZMA provided the best compression at the cost of reading speed. The compression ratio was of 10.63084:1 and the reading speed was 14.1 ms ± 1.37ms per loop (mean ± std. dev. of 7 runs, 100 loops each).

• GZIP provided intermediate compression rate of 10.63084:1 and moderate reading speed of 5.21 ms ± 351µs per loop (mean ± std. dev. of 7 runs, 100 loops each).

• BZ2 provided the best compression rate (11.32589:1) and the slowest reading speed, where reading a file took 30.3 ms ± 5.66ms per loop (mean ± std. dev. of 7 runs, 10 loops each).

• brotli provided the third best compression ratio (9.73374:1) and the second highest reading speed with 3.46 ms ± 249µs per loop (mean ± std. dev. of 7 runs, 100 loops each), which I think will fit many people's use needs. In my case, the reading speed was an important factor, so the difference with other compression methods made me stray away from this particular compression algorithm for my specific use case.

• ZFP did not have an implementation to compress float16 data so the compression factor was not great (3.16857:1). Moreover, at a reading speed of 10 ms ± 781µs per loop (mean ± std. dev. of 7 runs, 100 loops each), this method was discarded for my application.

• Finally, ZSTD provided the third best compression rate (8.64682:1) and, by far, the fastest reading speed of 1.03 ms ± 16.6µs per loop (mean ± std. dev. of 7 runs, 1000 loops each). This is the method that I chose for my application.

Hope it helps