I've been using HDF5 for years, but as the size of the dataset grows I'm starting to experience the same problems listed here


Can you point me to a format like HDF5 with - Good support for parallel writes - Support for chunked access of huge matrices

My typical use case is a 100k x 100k integer matrix. I'd like to have it as a whole file from a logical perspective, but I need to write it chunk by chunk with parallel workers.

  • 3
    $\begingroup$ Can you please explain what kinds of datasets you need to export? This may be helpful to people looking to answer your question. I've considered HDF5 as well as netcfd. But these may be more geared to certain data sets. $\endgroup$ – Charles Oct 15 '16 at 3:19
  • 2
    $\begingroup$ Compressed VTK supports chunks. You can save parallel many files and merge it together using PVD meta file. What is the size of your dataset? $\endgroup$ – Krzysztof Bzowski Oct 15 '16 at 7:09
  • $\begingroup$ After your edits - chunked VTK is not a good decision. $\endgroup$ – Krzysztof Bzowski Oct 15 '16 at 7:54
  • $\begingroup$ What did you end up using? $\endgroup$ – aidan.plenert.macdonald Apr 13 '17 at 21:57
  • $\begingroup$ @aidan.penert.macdonald I kept with hdf5, using parallel writes with MPI. But I had to abandon Python $\endgroup$ – M. G. Apr 19 '17 at 16:09

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.