I've been using HDF5 for years, but as the size of the dataset grows I'm starting to experience the same problems listed here


Can you point me to a format like HDF5 with - Good support for parallel writes - Support for chunked access of huge matrices

My typical use case is a 100k x 100k integer matrix. I'd like to have it as a whole file from a logical perspective, but I need to write it chunk by chunk with parallel workers.

  • 3
    $\begingroup$ Can you please explain what kinds of datasets you need to export? This may be helpful to people looking to answer your question. I've considered HDF5 as well as netcfd. But these may be more geared to certain data sets. $\endgroup$
    – Charles
    Oct 15 '16 at 3:19
  • 2
    $\begingroup$ Compressed VTK supports chunks. You can save parallel many files and merge it together using PVD meta file. What is the size of your dataset? $\endgroup$ Oct 15 '16 at 7:09
  • $\begingroup$ After your edits - chunked VTK is not a good decision. $\endgroup$ Oct 15 '16 at 7:54
  • $\begingroup$ What did you end up using? $\endgroup$
    – ignorance
    Apr 13 '17 at 21:57
  • $\begingroup$ @aidan.penert.macdonald I kept with hdf5, using parallel writes with MPI. But I had to abandon Python $\endgroup$
    – M. G.
    Apr 19 '17 at 16:09

HDF5 is, to some extent, a filesystem on its own. By introducing B-Trees and by the way it manages blocks, it duplicates the functionality of a filesystem. When you are running your code, you are probably running it on an operating system with a proven and scalable filesystem. Hence, I would suggest to write your numerical raw data into a single file using raw file access or MPI-IO and write the meta-data (endianess, size, attributes, etc.) into a separate JSON or XML file. If you have multiple datasets you can organize them into a directory or a hierarchy of directories. When you want to distribute the dataset, you just have to pack it into a ZIP file.

The only downside is that you have to deal with Endianness yourself, which is, however, not hard.

For an inspiration on how this can be done see Dragly, et. al. "A. Experimental Directory Structure (Exdir): An Alternative to HDF5 Without Introducing a New File Format" Front. Neuroinform., 2018, 12.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.