5
$\begingroup$

Reposted from: https://stackoverflow.com/q/34909093/472610

I work in scientific research. Technical tools aren't always ideal or even really thought through sometimes, but with recent developments one recurrent need (in my field and many other I know of) is that of storing reasonably large datasets (10GB - 1TB) with large amounts of meta-data in a reliable and efficient way. I'm not interested in distributed storage at all here.

Many ad hoc file formats with variably well-defined specifications exist. Datasets become naturally more complex over time, or integrate data coming from different fields with their own file-formats, which usually results either a big (inefficient and error-prone) mess or new ad hoc file-formats.

Many people have chosen to turn to generic storage formats specifically designed for large datasets by people who know what they are doing, like HDF5. The feedback on this experience is mitigated, and many people criticise especially the complexity of designing an efficient structure for their dataset, and the fact that crashes during processing tend to result in data corruption, which in turn means losing the entire dataset.

I don't know very many things about file-systems, but I have some experience with virtual machines and one thing that blew my mind the first time I created one is that I had to create a virtual hard drive, which would simply sit as a big file in my own file-system. Even better, there was the option of creating a resizeable one, so it could grow as I needed more and more memory.

My questions are, to people who know these things:

  • Would there be non-trivial disadvantages in simply storing large datasets as their own file structure in a virtual file system? It would be platform independent, wouldn't it?
  • Would it be possible to choose a file-system with built-in redundancy or checksum features, such that it could deal with data corruption regardless of the host system (eg ZFS)? Are there portable (or at least unix/linux friendly), open libraries, to create/access/modify such virtual filesystems?
  • Would it be possible to mount such drives easily, say as an external drive (just as an .iso file for example)? Would you know of any portable & open source tool to do this?
Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
1
$\begingroup$

What you describe sounds like a way where your whole data set is stored in one directory with a set of subdirectories and files within them.

If you think about this differently, then you will come to realize that that's exactly how an XML file is structured: there are hierarchies of sections (subdirectories) and some tags have values (files). In some cases, the values are compressed (e.g., uuencoded, bzip-compressed data) which corresponds to compressing individual files.

In other words, what you suggest already exists in hierarchically structured file formats such as XML. The difference simply is that you don't have hundreds or thousands of files sitting around requiring their own metadata, but it's all stored in one file. That's a bit like zipping up the whole directory and compressing it. In essence, that's also what .docx or the OpenOffice file formats do: they're just compressed, tar'ed (or zipped) directory structures.

Improve this answer
$\endgroup$
4
  • $\begingroup$ There is a plethora of formats (including HDF5) that store data in a tree-like structure, XML being arguably the canonical one. The advantage of using a file-system instead of a structured file though, is that you get the many amazing features that modern file-systems provide for free; mainly fast-access to subfields without need to load the whole data into RAM / mapping the memory on disk manually, and robustness to corruption (by journaling, checksum, error codes, etc) with the worst case being some parts of the data becoming inaccessible as opposed to the entire file being unreadable. $\endgroup$
    – Jonathan H
    Jan 28, 2016 at 18:41
  • $\begingroup$ You pay for that by having to go through the operating system and writing data into so many different places, though. Besides, offset tables, checkpointing, etc, can of course all be emulated by parts of the data files themselves. $\endgroup$
    – Wolfgang Bangerth
    Jan 28, 2016 at 21:57
  • $\begingroup$ I don't understand what you mean; this doesn't require an operating system on the virtual disk, file-systems are independent of what runs on them. If you look at what actually happens, then yes it is way more complex than an XML file, but this complexity is transparent as a user, and you get extras like symbolic links, the ability to take lightweight snapshots of your dataset, etc. Again the question is not whether you could do the same manually, the question is why bother? $\endgroup$
    – Jonathan H
    Jan 29, 2016 at 0:54
  • $\begingroup$ But it's no more complicated to use a good XML library, or the HDF5 library. It just all happens in user memory. $\endgroup$
    – Wolfgang Bangerth
    Jan 29, 2016 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.