# Compressing floating point data

Are there any tools specifically designed for compressing floating point scientific data?

If a function is smooth, there's obviously a lot of correlation between the numbers representing that function, so the data should compress well. Zipping/gzipping binary floating point data doesn't compress it that well though. I am wondering if there are method specifically developed for compressing floating point data.

Requirements:

• Either lossless compression or the possibility to specify a minimum number of digits to retain (for some applications double might be more than what we need while float might not have enough precision).

• Well tested working tool (i.e. not just a paper describing a theoretical method).

• Suitable for compressing 1D numerical data (such as a time series)

• Cross platform (must work on Windows)

• It must be fast---preferably not much slower than gzip. I found that if I have the numbers stored as ASCII, gzipping the file can speed up reading and processing it (as the operation might be I/O bound).

I'd especially like to hear from people who have actually used such a tool.

• This was partly inspired by the existence of FLAC, which suggests that a specialized method should do (much?) better than gzip. – Szabolcs Mar 18 '12 at 10:27
• I'm looking at this now. – Szabolcs Mar 19 '12 at 7:56
• Neat. I am going to give this one a whirl. – meawoppl Feb 22 '13 at 19:39

Try out Blosc. It is in many cases faster than memcopy. Think about that for a second. . . wicked.

It is super stable, highly-vetted, cross-platform, and performs like a champ.

• oh wow, this is really cool (and new to me!) – Aron Ahmadia Mar 18 '12 at 7:22
• The link is broken. Any chance you'd know where it is now? – Alexis Wilke Feb 2 '16 at 0:58
• @AlexisWilke I fixed the link. It was the first result in a google search for Blosc. – Doug Lipinski Feb 4 '16 at 17:16
• Blosc is maybe fast but its compression rate on float arrays is a disaster. With the best compression it offers it results in about 98% of the original size. Thanks for the tip in any case. – user20600 Jun 5 '16 at 12:57
• Compression on float arrays depends heavily on the content. I suspect there is little (structured) information in the bits you are compressing. Also, blosc is still under active dev 5 years later! – meawoppl Sep 16 '17 at 15:25

I got good results using HDF5 and its GZIP filter.

The HDF5 also provides an SZIP filter which achieves better results for some scientifica data-sets.

In my experience the choice of compressions depends heavily on the kind of data and benchmarking is probably the only way to make a good choice.

BTW, third-party filters for HDF5 include BLOSC, BZIP2, LZO, LZF, MAFISC.

• Thanks fot the answer! I have not used HDF5 much. Is it correct that using the gzip filter with the HDF5 format is going to give me the same compression ratio as writing all the number into a flat binary file and running it through gzip? (Ignore the possible convenience/inconvenience of using HDF5 for now.) Regarding SZIP, is it in some way optimized for floating point datasets? (I'm curious and this is not clear from skimming the page you linked.) The page says the primary advantage of SZIP is speed. GZIP is also pretty snappy (usually gzip decompression takes negligible for me). – Szabolcs May 24 '12 at 14:33
• A gzipped flat binary file will probably be smaller than a HDF5 file with gzip filter, because HDF5 is more than raw data. Sometimes preprocessing with a shuffle filter can improve the gzip results. But you are right, the advantages are indeed more rather convenience. With HDF5 I find it easy to change the compression filter (try out different settings) and HDF5 provides function to acces subsets of your data (intervals in time series). – f3lix May 25 '12 at 7:34
• If you go this route check out pyTables. It makes the above just a couple lines of code. Maintained (previously at least) by the Blosc author. – meawoppl Feb 22 '13 at 19:36

Arguably, you can interpret regression or transform methods (Fourier transform, Chebyshev transform) methods as "compression" for time-series or 1D function data. Remez's algorithm would be another candidate. In that case, using something like regression, FFT, or Chebyshev via FFT would work for your purposes. That said, none of these methods works on time series data with arbitrary structure. For example, with FFT, you assume periodicity, and any sort of discontinuities in the data (or lack of periodicity) will lead to Gibbs' phenomenon. Similarly, with Chebyshev transforms, the assumption is that the data describes a function on $[-1,1]$.

Depending on the underlying function, you may be able to fit the data to a functional form without error, requiring fewer coefficients to describe the functional form than you have data point (leading to the compression). Error results exist for some of these methods, although I don't know if any of them will give you a priori (or a posteriori) bounds or estimates on the error.

You could also look at methods developed specifically for the compression of floating point numbers, like FPC and related algorithms. See the papers here, here, here, here, and here, along with a web page containing old source code here.

• Actually I am interested in ready-made tools similar to gzip that do not require any work on my part, especially not developing and tuning my own method. Also, it'd be advantageous to have a method which doesn't require reading the whole thing into memory before decompressing it as I might have very large datafiles that can be processed sequentially (this works with gzip, but not if I use a Fourier transform, unless I slice the data into chunks myself, complicating the whole thing even more) Something that assumes that my datafile is just a series of binary doubles would be excellent. – Szabolcs Mar 17 '12 at 20:11
• Also these are 1:1 transformations not really compression techniques. They can be used to create data that a naive compression algorithm can do better with, but standalone not a solution. – meawoppl Feb 22 '13 at 19:38
• Some of these methods form the mathematical basis for compression algorithms used in signal processing, which was the idea behind the answer. These transformations typically aren't 1:1 except under special circumstances. – Geoff Oxberry Feb 22 '13 at 22:43

HDF5 can use a "shuffling" algorithm where the bytes for N floating point numbers are rearranged so that the first bytes of the N numbers come first, then the 2nd, and so on. This produces better compression ratios after gzip is applied, as it is more likely to produce longer sequences of the same value. See here for some benchmarks.

SZ (developed by Argonne in 2016) could be a good choice.

SZ: Fast Error-Bounded Floating-point Data Compressor for Scientific Applications https://collab.cels.anl.gov/display/ESR/SZ

• Why do you think it could be a good choice? What are its capabilities in comparison to other compression techniques? – Paul Jan 13 '17 at 22:45

Possible methods, that can be used for floating-point compression:

• Transpose 4xN for float and 8xN for double + lz77
Implementation: Floating point compression in TurboTranspose

• Predictor (ex. Finite Context Method) + encoding (ex. "integer compression").
Implementation: Floating point compression in TurboPFor
including special compression for time series.

• when possible, convert all floating point numbers to integers (ex. 1.63 -> 163), then use integer compression

• You can test all these methods with your data using the icapp tool for linux and windows.

We have been using ZFP with HDF5 for our medical imaging data. It is made for lossy, floating point compression.

We're running it on literally everything, and have more than 40TB of data stored (and being used!). It is fast enough to save our data real-time, and we can specify the required precision, so while the format is lossy, we're not seeing any differences in our final outputs.

If a function is smooth, there's obviously a lot of correlation between the numbers representing that function, so the data should compress well.

Perhaps the format you require need to store just the offsets from value to neighboring value.

Alternately, maybe you could make use of the frequency domain, perhaps even saving these values as a lossless audio file such as "flac lossless", as you require some of the same properties to a sound.

However, I'm going to take a different approach to attempting to answer the question which I hope can be of some help. As what you are saying is also that the minimum description length to represent this data is less than providing all the data points.

https://en.wikipedia.org/wiki/Minimum_description_length

Effectively a program, computer code, is a good example. And if you don't mind that somthing thats primarily data working by executing, and so also being code, then you could compress your floating point values into somthing like a function or formuli.

Doing this particularly well automatically, and in a realistic amount of compute, is beyond hard. However the Wolfram Language provides some functionality to attempt this:

https://reference.wolfram.com/language/ref/RSolve.html

Why not just save float32 / float16 ? In numpy,

A.astype( np.float32 )  # 100M: 200 msec imac
A.astype( np.float16 )  # 100M: 700 msec


These won't do if you're simulating the Butterfly effect in chaos theory, but they're understandable, portable, "do not require any work on my part". And compression 2:1 / 4:1 over float64 is hard to beat :)

Notes:

"Array type float16 is unsupported in np.linalg"; you'll have to expand it to 32 or 64 after reading it in.

To see how floating-point parameters differ,

import numpy as np
for f in [np.float64, np.float32, np.float16]:
print np.finfo(f)


For a plot of a trivial test case comparing float 64 32 and 16, see here .