1
$\begingroup$

When running my simulations I usually get result files with typical sizes of 10 GByte and larger. They contain vectors and matrices with data type float, double, complex float and complex double.

Initially I used text files, but when storing the data for each simulation step in text files the amount of generated files increases quite fast. Thus, I switched to using HDF5. It has the following advantages:

  • I have all results in a single file, and not distributed as in text files
  • I can store attributes and metadata next to the stored data, describing the content and used parameters
  • I can structure my data, such that I can go through it like in a file system. Related data is kept close together, I do not have to pluck data for my x-axis in one place, my t-axis in a completely different place, and the corresponding matrix at a third place. This makes handling extremely simple
  • I can access the data via Matlab, Python, C/C++ and several other languages
  • I could (in theory) write data in parallel, but that is not necessary at the moment

However, this file format has several drawbacks (which do not exist with txt-files):

  • Opening a file without closing it properly will corrupt it, and can make it unusable (in the worst case)
  • Writing in a file and hitting the quota on the hard disk will corrupt the file, and make it completely unusable.

In short, it is extremely easy to corrupt the file, which can be especially painful if the corresponding simulation took several weeks to run, and no backup is available.

What other alternatives do I have with the advantages listed here, and being more robust than HDF5? This question might be related to others (Best practice for storing hierarchical simulation data or Alternatives to hdf5), and there alternatives such as XML or ADIOS are proposed, but I have not found a final conclusion yet.

Improve this question
$\endgroup$
4
  • $\begingroup$ Thanks for providing info on the data types but can you add a bit about the structure of your data? Is it a set of spatial values that are time dependent? Is it hierarchical? This will help to suggest some other options. $\endgroup$
    – Kyle Mandli
    Dec 20, 2021 at 16:00
  • $\begingroup$ I am doing steps in time in my simulation, and at some steps I store only vectors, and at others I store both vectors and matrices (both supported by vectors representing coordinate data). Currently I separate both into "cheap" and "expensive" subgroups. Does that information help? $\endgroup$
    – arc_lupus
    Dec 20, 2021 at 16:17
  • $\begingroup$ NetCDF won't work for you then unfortunately as it the simplification over HDF5 it makes have to do with static dimensions that scale infinitely in time so the different sizes will make this unlikely to work well. $\endgroup$
    – Kyle Mandli
    Dec 21, 2021 at 20:04
  • $\begingroup$ also HDF5 has global lock, so unable to open and read multiple datasets simultaneously from single process $\endgroup$
    – iperov
    Jan 4 at 17:15

2 Answers 2

Reset to default
3
$\begingroup$

I think you put too much weight on your list of disadvantages. If you run up to the disk space limit, for example, does it really make a difference whether (i) half of the data is lost simply because it could not be written, or (ii) all of the data is lost because the file is corrupted? In either case, you do not have the data for which you ran the simulation, and it will have to be run again. If you anticipate running out of disk space being a problem and made sure that the "most important parts of the data" are written first into the .txt file, you can emulate the same by using more than one .h5 file, but in practice I imagine that you cannot reasonably deal with the situation of running out of disk space because you cannot anticipate which parts of the data will be written and for which parts the write will fail: It's just a fragile approach that will turn out not be very useful in practice.

"Opening a file without properly closing it" is a problem with any kind of file. Both the programming language support libraries and the operating system will always buffer writes, and whatever you read from a file that has not been closed properly is unreliable in any case. So you should use programming practices that make that impossible. For example, in C++ you would use classes that close the file when you leave a code block (e.g., the std::ofstream class via its destructor) and you should use similar classes for writing to HDF5 files. The same kinds of mechanisms are available in all modern programming languages (to which I will not count C or the variations of Fortran most people today use).

In other words, I believe that the advantages you give to HDF5 are all real, whereas the disadvantages are not.

Improve this answer
$\endgroup$
6
  • $\begingroup$ I understand your reasoning. Concerning the first point: For me it would make a difference if I can still read 50 % of my data, or nothing at all. If I still can read 50 % of my data I can use it as new starting point, saving me several days to week of calculation time (after checking for corruption/errors). Concerning the second point: I have not experienced any issues with corrupted text files yet when opening them with open(file_name, 'r') and never closing them correctly, but with hdf5-files I have. Therefore, I would like to avoid/circumvent those issues as much as possible. $\endgroup$
    – arc_lupus
    Dec 21, 2021 at 8:08
  • $\begingroup$ @arc_lupus Then you just got lucky. As for the first point: But you can't rely on it being 50% of the data. It could be 30% or it could be 70%. You just can't know. If you need to rely on smaller increments, you need to write more and smaller files. $\endgroup$
    – Wolfgang Bangerth
    Dec 21, 2021 at 17:17
  • $\begingroup$ Of course, I can't be sure if I got 10% of my steps, 20% of my steps or 90%. I am writing my steps in order, and therefore the assumption was that if I have at least one step in my result file, I can verify if the data for this step is complete and correct. If I only save half the data for a step, I'll be able to just discard it, but can still use the data from the full steps. $\endgroup$
    – arc_lupus
    Dec 21, 2021 at 19:46
  • 1
    $\begingroup$ I think as @WolfgangBangerth pointed out that this is more of a programming problem to be coded around in a robust way rather than a failure of HDF5 or any other high performance file format. $\endgroup$
    – Kyle Mandli
    Dec 21, 2021 at 20:05
  • $\begingroup$ @arc_lupus I might have misunderstood, but are you saying that you're putting all time steps into one HDF5 file? But that you are putting one time step into each .txt file? Well, there's your problem. $\endgroup$
    – Wolfgang Bangerth
    Dec 21, 2021 at 22:35
1
$\begingroup$

Did you take a look into (Parallel) NetCDF? I am not sure, however, whether there is a handling for the issues that you mentioned which is guaranteed on all platforms and interfaces.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ NetCDF is built on top of HDF5 except with an assumed structure in space time so I highly doubt that this will be any better along the lines of the original question. $\endgroup$
    – Kyle Mandli
    Dec 20, 2021 at 11:22
  • 1
    $\begingroup$ I should also add that this is only the case for NetCDF4 although I would not suggest using something older at this point. $\endgroup$
    – Kyle Mandli
    Dec 20, 2021 at 16:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.