Are there any tools specifically designed for compressing floating point scientific data?

If a function is smooth, there's obviously a lot of correlation between the numbers representing that function, so the data should compress well. Zipping/gzipping binary floating point data doesn't compress it that well though. I am wondering if there are methods specifically developed for compressing floating point data.


  • Either lossless compression or the possibility to specify a minimum number of digits to retain (for some applications double might be more than what we need while float might not have enough precision).

  • Well tested working tool (i.e. not just a paper describing a theoretical method).

  • Suitable for compressing 1D numerical data (such as a time series)

  • Cross platform (must work on Windows)

  • It must be fast---preferably not much slower than gzip. I found that if I have the numbers stored as ASCII, gzipping the file can speed up reading and processing it (as the operation might be I/O bound).

I'd especially like to hear from people who have actually used such a tool.

  • 1
    $\begingroup$ This was partly inspired by the existence of FLAC, which suggests that a specialized method should do (much?) better than gzip. $\endgroup$
    – Szabolcs
    Commented Mar 18, 2012 at 10:27
  • 1
    $\begingroup$ I'm looking at this now. $\endgroup$
    – Szabolcs
    Commented Mar 19, 2012 at 7:56
  • $\begingroup$ Neat. I am going to give this one a whirl. $\endgroup$
    – meawoppl
    Commented Feb 22, 2013 at 19:39

10 Answers 10


Try out Blosc. It is in many cases faster than memcopy. Think about that for a second. . . wicked.

It is super stable, highly-vetted, cross-platform, and performs like a champ.

  • $\begingroup$ oh wow, this is really cool (and new to me!) $\endgroup$ Commented Mar 18, 2012 at 7:22
  • $\begingroup$ The link is broken. Any chance you'd know where it is now? $\endgroup$ Commented Feb 2, 2016 at 0:58
  • 1
    $\begingroup$ @AlexisWilke I fixed the link. It was the first result in a google search for Blosc. $\endgroup$ Commented Feb 4, 2016 at 17:16
  • 2
    $\begingroup$ Blosc is maybe fast but its compression rate on float arrays is a disaster. With the best compression it offers it results in about 98% of the original size. Thanks for the tip in any case. $\endgroup$
    – user20600
    Commented Jun 5, 2016 at 12:57
  • $\begingroup$ Compression on float arrays depends heavily on the content. I suspect there is little (structured) information in the bits you are compressing. Also, blosc is still under active dev 5 years later! $\endgroup$
    – meawoppl
    Commented Sep 16, 2017 at 15:25

I got good results using HDF5 and its GZIP filter.

The HDF5 also provides an SZIP filter which achieves better results for some scientifica data-sets.

In my experience the choice of compressions depends heavily on the kind of data and benchmarking is probably the only way to make a good choice.

BTW, third-party filters for HDF5 include BLOSC, BZIP2, LZO, LZF, MAFISC.

  • $\begingroup$ Thanks fot the answer! I have not used HDF5 much. Is it correct that using the gzip filter with the HDF5 format is going to give me the same compression ratio as writing all the number into a flat binary file and running it through gzip? (Ignore the possible convenience/inconvenience of using HDF5 for now.) Regarding SZIP, is it in some way optimized for floating point datasets? (I'm curious and this is not clear from skimming the page you linked.) The page says the primary advantage of SZIP is speed. GZIP is also pretty snappy (usually gzip decompression takes negligible for me). $\endgroup$
    – Szabolcs
    Commented May 24, 2012 at 14:33
  • $\begingroup$ A gzipped flat binary file will probably be smaller than a HDF5 file with gzip filter, because HDF5 is more than raw data. Sometimes preprocessing with a shuffle filter can improve the gzip results. But you are right, the advantages are indeed more rather convenience. With HDF5 I find it easy to change the compression filter (try out different settings) and HDF5 provides function to acces subsets of your data (intervals in time series). $\endgroup$
    – f3lix
    Commented May 25, 2012 at 7:34
  • 1
    $\begingroup$ If you go this route check out pyTables. It makes the above just a couple lines of code. Maintained (previously at least) by the Blosc author. $\endgroup$
    – meawoppl
    Commented Feb 22, 2013 at 19:36

Arguably, you can interpret regression or transform methods (Fourier transform, Chebyshev transform) methods as "compression" for time-series or 1D function data. Remez's algorithm would be another candidate. In that case, using something like regression, FFT, or Chebyshev via FFT would work for your purposes. That said, none of these methods works on time series data with arbitrary structure. For example, with FFT, you assume periodicity, and any sort of discontinuities in the data (or lack of periodicity) will lead to Gibbs' phenomenon. Similarly, with Chebyshev transforms, the assumption is that the data describes a function on $[-1,1]$.

Depending on the underlying function, you may be able to fit the data to a functional form without error, requiring fewer coefficients to describe the functional form than you have data point (leading to the compression). Error results exist for some of these methods, although I don't know if any of them will give you a priori (or a posteriori) bounds or estimates on the error.

You could also look at methods developed specifically for the compression of floating point numbers, like FPC and related algorithms. See the papers here, here, here, here, and here, along with a web page containing old source code here.

  • $\begingroup$ Actually I am interested in ready-made tools similar to gzip that do not require any work on my part, especially not developing and tuning my own method. Also, it'd be advantageous to have a method which doesn't require reading the whole thing into memory before decompressing it as I might have very large datafiles that can be processed sequentially (this works with gzip, but not if I use a Fourier transform, unless I slice the data into chunks myself, complicating the whole thing even more) Something that assumes that my datafile is just a series of binary doubles would be excellent. $\endgroup$
    – Szabolcs
    Commented Mar 17, 2012 at 20:11
  • $\begingroup$ Also these are 1:1 transformations not really compression techniques. They can be used to create data that a naive compression algorithm can do better with, but standalone not a solution. $\endgroup$
    – meawoppl
    Commented Feb 22, 2013 at 19:38
  • $\begingroup$ Some of these methods form the mathematical basis for compression algorithms used in signal processing, which was the idea behind the answer. These transformations typically aren't 1:1 except under special circumstances. $\endgroup$ Commented Feb 22, 2013 at 22:43

HDF5 can use a "shuffling" algorithm where the bytes for N floating point numbers are rearranged so that the first bytes of the N numbers come first, then the 2nd, and so on. This produces better compression ratios after gzip is applied, as it is more likely to produce longer sequences of the same value. See here for some benchmarks.


We have been using ZFP with HDF5 for our medical imaging data. It is made for lossy, floating point compression.

We're running it on literally everything, and have more than 40TB of data stored (and being used!). It is fast enough to save our data real-time, and we can specify the required precision, so while the format is lossy, we're not seeing any differences in our final outputs.


SZ (developed by Argonne in 2016) could be a good choice.

SZ: Fast Error-Bounded Floating-point Data Compressor for Scientific Applications https://collab.cels.anl.gov/display/ESR/SZ

  • $\begingroup$ Why do you think it could be a good choice? What are its capabilities in comparison to other compression techniques? $\endgroup$
    – Paul
    Commented Jan 13, 2017 at 22:45

Possible methods, that can be used for floating-point compression:

  • Transpose 4xN for float and 8xN for double + lz77
    Implementation: Floating point compression in TurboTranspose
    see also error-bounded lossy compression

  • Predictor (ex. Finite Context Method) + encoding (ex. "integer compression").
    Implementation: Floating point compression in TurboPFor
    including special compression for time series.

  • when possible, convert all floating point numbers to integers (ex. 1.63 -> 163), then use integer compression

  • You can test all these methods with your data using the icapp tool for linux and windows.


If a function is smooth, there's obviously a lot of correlation between the numbers representing that function, so the data should compress well.

Perhaps the format you require need to store just the offsets from value to neighboring value.

Alternately, maybe you could make use of the frequency domain, perhaps even saving these values as a lossless audio file such as "flac lossless", as you require some of the same properties to a sound.

However, I'm going to take a different approach to attempting to answer the question which I hope can be of some help. As what you are saying is also that the minimum description length to represent this data is less than providing all the data points.


Effectively a program, computer code, is a good example. And if you don't mind that somthing thats primarily data working by executing, and so also being code, then you could compress your floating point values into somthing like a function or formuli.

Doing this particularly well automatically, and in a realistic amount of compute, is beyond hard. However the Wolfram Language provides some functionality to attempt this:

https://reference.wolfram.com/language/ref/FindSequenceFunction.html https://reference.wolfram.com/language/ref/FindGeneratingFunction.html https://reference.wolfram.com/language/ref/FindFormula.html



Why not just save float32 / float16 ? In numpy,

A.astype( np.float32 )  # 100M: 200 msec imac
A.astype( np.float16 )  # 100M: 700 msec

These won't do if you're simulating the Butterfly effect in chaos theory, but they're understandable, portable, "do not require any work on my part". And compression 2:1 / 4:1 over float64 is hard to beat :)


"Array type float16 is unsupported in np.linalg"; you'll have to expand it to 32 or 64 after reading it in.

To see how floating-point parameters differ,

import numpy as np
for f in [np.float64, np.float32, np.float16]:
    print np.finfo(f)

For a plot of a trivial test case comparing float 64 32 and 16, see here .


This is one of the first results that come up when you search in google. My use case is whole slide images (i.e. huge TIFF files with JPEG compression) over which I had to perform some post-processing and save the results. So I have huge images (e.g. 100000 x 90000 pixels), 2 channels w/. float16 information.

For my specific application it made sense to divide the image in patches, so I tiled the image in 512 x 512 pixel patches. In this setting, three compression methods stood out:

  • LZMA provided the best compression at the cost of reading speed. The compression ratio was of 10.63084:1 and the reading speed was 14.1 ms ± 1.37ms per loop (mean ± std. dev. of 7 runs, 100 loops each).

  • GZIP provided intermediate compression rate of 10.63084:1 and moderate reading speed of 5.21 ms ± 351µs per loop (mean ± std. dev. of 7 runs, 100 loops each).

  • BZ2 provided the best compression rate (11.32589:1) and the slowest reading speed, where reading a file took 30.3 ms ± 5.66ms per loop (mean ± std. dev. of 7 runs, 10 loops each).

  • brotli provided the third best compression ratio (9.73374:1) and the second highest reading speed with 3.46 ms ± 249µs per loop (mean ± std. dev. of 7 runs, 100 loops each), which I think will fit many people's use needs. In my case, the reading speed was an important factor, so the difference with other compression methods made me stray away from this particular compression algorithm for my specific use case.

  • ZFP did not have an implementation to compress float16 data so the compression factor was not great (3.16857:1). Moreover, at a reading speed of 10 ms ± 781µs per loop (mean ± std. dev. of 7 runs, 100 loops each), this method was discarded for my application.

  • Finally, ZSTD provided the third best compression rate (8.64682:1) and, by far, the fastest reading speed of 1.03 ms ± 16.6µs per loop (mean ± std. dev. of 7 runs, 1000 loops each). This is the method that I chose for my application.

Hope it helps


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.