When running my simulations I usually get result files with typical sizes of 10 GByte and larger. They contain vectors and matrices with data type float, double, complex float and complex double.
Initially I used text files, but when storing the data for each simulation step in text files the amount of generated files increases quite fast. Thus, I switched to using HDF5. It has the following advantages:
- I have all results in a single file, and not distributed as in text files
- I can store attributes and metadata next to the stored data, describing the content and used parameters
- I can structure my data, such that I can go through it like in a file system. Related data is kept close together, I do not have to pluck data for my
x-axisin one place, my
t-axisin a completely different place, and the corresponding matrix at a third place. This makes handling extremely simple
- I can access the data via Matlab, Python, C/C++ and several other languages
- I could (in theory) write data in parallel, but that is not necessary at the moment
However, this file format has several drawbacks (which do not exist with txt-files):
- Opening a file without closing it properly will corrupt it, and can make it unusable (in the worst case)
- Writing in a file and hitting the quota on the hard disk will corrupt the file, and make it completely unusable.
In short, it is extremely easy to corrupt the file, which can be especially painful if the corresponding simulation took several weeks to run, and no backup is available.
What other alternatives do I have with the advantages listed here, and being more robust than HDF5? This question might be related to others (Best practice for storing hierarchical simulation data or Alternatives to hdf5), and there alternatives such as XML or ADIOS are proposed, but I have not found a final conclusion yet.