I am not familiar with the file format used in HDF5, but I am wondering if HDF5 files are suitable for revision control with git (or for example Mercurial or Subversion)? I guess what I mean is: are HDF5 files suitable for line-based diff'ing or will git have to treat an HDF5 as one big binary and store an entire copy for each revision?
$\begingroup$
$\endgroup$
6
-
1$\begingroup$ HDF5 is designed for binary data. They aren't really appropriate for line diffing. That being said, if all you write to them is ASCII strings, you'll probably mostly get away with it. What's your purpose? $\endgroup$– Bill BarthSep 18, 2013 at 11:53
-
$\begingroup$ I was just wondering if they would be suitable for revision control. It gets inconvenient if the revision tracking has to store an entire new copy of the entire data set each time a relatively small change has been made to it. $\endgroup$– Thomas ArildsenSep 18, 2013 at 12:07
-
1$\begingroup$ What kinds of data were you planning on putting in your HDF5 files? HDF5 files are typically used for large binary inputs and outputs from simulation codes. The former often don't change frequently, and it's not clear that the latter belong in revision control. What's your goal? $\endgroup$– Bill BarthSep 18, 2013 at 12:15
-
$\begingroup$ I am thinking about situations such as discarding data entries from your data set due to quality control or adding additional data to sets of already existing data. $\endgroup$– Thomas ArildsenSep 18, 2013 at 12:54
-
2$\begingroup$ HDF5 probably won't diff well, but you have to ask yourself which is more important to you: the size of your repo, or the features that HDF5 gives you. Perhaps a better question would be "What's the best way to store raw data that provides version history and provenance features?" $\endgroup$– Bill BarthSep 18, 2013 at 13:52
|
Show 1 more comment
3 Answers
$\begingroup$
$\endgroup$
2
You're going to get a much better answer if you provide a few more technical details about what kind of data you're trying to put under version control, how you want to store different versions of the data, what components are likely to change and what components aren't, and whether you're truly going to have tree-like history (branches, merges).
HDF5 files are not suitable for diff-based version control under git.
git uses a hash-based database under the hood, so it is possible to store the hash of your HDF5 data file without actually storing the file itself. Three projects, git-fat, git-annex and git-media, greatly simplify this process for you. I would suggest using this approach if you have large, completely independent chunks of data which you would like to explicitly version.
If you can separate your data storage into non-volatile and volatile regions, this will greatly improve the efficiency of your interaction with the version control database. You may also want to consider explicitly using a database for your data if you do not need the DVCS features git offers.
answered Sep 19, 2013 at 17:10
-
$\begingroup$ It's also possible to version control databases, if that's what you want to do, by version controlling the schema, dumping the database to a text file, and version controlling the result (e.g., using git). See stackoverflow.com/questions/846659/… for details. $\endgroup$ Sep 19, 2013 at 20:57
-
$\begingroup$
$\endgroup$
I guess what I mean is: are HDF5 files suitable for line-based diff'ing or will git have to treat an HDF5 as one big binary and store an entire copy for each revision?
The literal answer to this question is that git will not treat HDF5 files efficiently.
For more useful answers about version control for projects that have some binary files, see this stackoverflow question: https://stackoverflow.com/questions/540535/managing-large-binary-files-with-git
$\begingroup$
$\endgroup$
2
As others said, it would be easier to make useful suggestions if you described your overall goal rather than a precise technical point. Here is one more suggestion that might help you, depending on what your goal is.
The ActivePapers project (http://www.activepapers.org/) provides a code and data management system on top of HDF5. An ActivePaper is an HDF5 file that contains datasets AND the code that works on them, with metadata keeping track of which piece of code computed which dataset and using which input data. In combination with version control on source code and/or version control on the whole HDF5 file (using tools such as git-annex, mentioned in another reply), ActivePapers can be used for versioning computations rather than isolated files or datasets.
Disclaimer: I am the author of ActivePapers.
-
1$\begingroup$ I am not currently working on a specific problem, but I was imagining some data set that you might be adding new data to from time to time. With each addition you might have to store an entire copy of the whole data set, which could be very large, while in principle, it would only be necessary to store a "diff" containing the added data. $\endgroup$ Jan 3, 2014 at 12:52
-
1$\begingroup$ I am not aware of any tool for doing diff/merge style operations on binary data, HDF5 or otherwise. One intriguing idea to do this with ActivePapers is to apply the change by including a "patch script" in the file along with the original data. You can then follow the evolution of the data as a sequence of applied patches. One advantage of the ActivePapers framework is that you can do the patches in a separate file the references the original. That means you can publish data, and publish modifications (to your own and someone else's data) later, as a separate work. $\endgroup$– khinsenJan 9, 2014 at 11:55