Reposted from: https://stackoverflow.com/q/34909093/472610
I work in scientific research. Technical tools aren't always ideal or even really thought through sometimes, but with recent developments one recurrent need (in my field and many other I know of) is that of storing reasonably large datasets (10GB - 1TB) with large amounts of meta-data in a reliable and efficient way. I'm not interested in distributed storage at all here.
Many ad hoc file formats with variably well-defined specifications exist. Datasets become naturally more complex over time, or integrate data coming from different fields with their own file-formats, which usually results either a big (inefficient and error-prone) mess or new ad hoc file-formats.
Many people have chosen to turn to generic storage formats specifically designed for large datasets by people who know what they are doing, like HDF5. The feedback on this experience is mitigated, and many people criticise especially the complexity of designing an efficient structure for their dataset, and the fact that crashes during processing tend to result in data corruption, which in turn means losing the entire dataset.
I don't know very many things about file-systems, but I have some experience with virtual machines and one thing that blew my mind the first time I created one is that I had to create a virtual hard drive, which would simply sit as a big file in my own file-system. Even better, there was the option of creating a resizeable one, so it could grow as I needed more and more memory.
My questions are, to people who know these things:
- Would there be non-trivial disadvantages in simply storing large datasets as their own file structure in a virtual file system? It would be platform independent, wouldn't it?
- Would it be possible to choose a file-system with built-in redundancy or checksum features, such that it could deal with data corruption regardless of the host system (eg ZFS)? Are there portable (or at least unix/linux friendly), open libraries, to create/access/modify such virtual filesystems?
- Would it be possible to mount such drives easily, say as an external drive (just as an
.iso
file for example)? Would you know of any portable & open source tool to do this?