9

You're going to get a much better answer if you provide a few more technical details about what kind of data you're trying to put under version control, how you want to store different versions of the data, what components are likely to change and what components aren't, and whether you're truly going to have tree-like history (branches, merges). HDF5 files ...


7

Purely size-wise, it is better to use binary. In ASCII representation, full expansion of a double is about 16+4 characters -depending on how you want to represent them size changes slightly- which is (at minimum) 20 bytes while in binary format it would take 8 bytes. That is more than half storage gain. Depending on how your binary formatting, I/O may also ...


7

HDF5 is, to some extent, a filesystem on its own. By introducing B-Trees and by the way it manages blocks, it duplicates the functionality of a filesystem. When you are running your code, you are probably running it on an operating system with a proven and scalable filesystem. Hence, I would suggest to write your numerical raw data into a single file using ...


4

I've rolled my own. Here is a MCVE: #include <Eigen/Core> #include <Eigen/Sparse> #include <iostream> #include <fstream> #include <vector> using namespace Eigen; typedef Triplet<int> Trip; template <typename T, int whatever, typename IND> void Serialize(SparseMatrix<T, whatever, IND>& m) { std::...


4

Let me start with the following: Even very good and experienced programmers have a very hard time estimating whether a particular piece of code is performance critical or not. This has given rise to the adage "Premature optimization is the root of all evil", which can be translated as "Unnecessarily optimizing code leads to obscure code that will be ...


3

I guess what I mean is: are HDF5 files suitable for line-based diff'ing or will git have to treat an HDF5 as one big binary and store an entire copy for each revision? The literal answer to this question is that git will not treat HDF5 files efficiently. For more useful answers about version control for projects that have some binary files, see this ...


3

As others said, it would be easier to make useful suggestions if you described your overall goal rather than a precise technical point. Here is one more suggestion that might help you, depending on what your goal is. The ActivePapers project (http://www.activepapers.org/) provides a code and data management system on top of HDF5. An ActivePaper is an HDF5 ...


3

HDF5 can use a "shuffling" algorithm where the bytes for N floating point numbers are rearranged so that the first bytes of the N numbers come first, then the 2nd, and so on. This produces better compression ratios after gzip is applied, as it is more likely to produce longer sequences of the same value. See here for some benchmarks.


3

Presumably you are using numpy arrays for your data. The key is to avoid making a copy when you reshape them. For that, you want to ensure that you are just generating new "views" of your arrays. Take a look at the help page for numpy.reshape.


3

I suggest taking a look at the FEAPpv finite element code written by Robert Taylor to accompany this book: http://www.amazon.com/The-Finite-Element-Method-Fundamentals/dp/1856176339 You can download the fortran source code from this page: http://www.ce.berkeley.edu/projects/feap/feappv/ The code contains two subroutines, datri and dasol that perform the ...


2

A relevant anecdote: Years ago I realized that the fellow who sat next to me in my church choir had played a role in defining the SGML standard. Sometime after that he started a conversation by asking, in a half-serious, sales-pitchy way, "In your experiment, do you store your data in a self-documenting text-based representation?" I thought about ...


2

If file size is an issue, you can use the zlib library functions gzopen, gzwrite, gzread, and gzclose to directly write and read the text to a compressed file. Apart from possibly saving even more space than writing numbers in binary format, it has the advantage that the file can be uncompressed from the command line with gzip to have a look at its content, ...


2

I will try answer based on my experience with deal.ii. The max_per_row=5 means that at most there will be 5 non-zeros per row in the matrix. Since we now know this then we do not need to have a $1\times{}100$ matrix but rather a $1\times{}50$. In other words this parameter sets an upper-bound on the memory needed. In reality it is not stored as one vector ...


2

You can use the CodeGeneration package to do this. It allows to translate to different languages, being Matlab one of those. A simple example here: with(CodeGeneration); suma := sum(sin(n*x)/factorial(n), n = 0 .. 10); then Matlab(suma) with answer cg3 = sin(x) + sin(0.2e1 * x) / 0.2e1 + sin(0.3e1 * x) / 0.6e1 + sin(0.4e1 * x) / 0.24e2 + sin(0.5e1 * x) ...


1

There is a very battle-tested library for this called fpzip, which has both lossless and lossy compression. There's a paper by the authors about their approach (here's a link without a paywall too). If you look at table 1 in their paper, they get compression ratios on the order of 100 for some simulation outputs, but as low as ~1.3 on others. Clearly the ...


1

What you describe sounds like a way where your whole data set is stored in one directory with a set of subdirectories and files within them. If you think about this differently, then you will come to realize that that's exactly how an XML file is structured: there are hierarchies of sections (subdirectories) and some tags have values (files). In some cases,...


1

The scipy.integrate.odeint function does not take a vector as input. It takes a function f, an initial state y0 and a vector of points in time t (and possibly a bunch of other optional arguments to control the integration strategy). So you will need to provide it with a function that calculates the right-hand-side of your ODE system, which will boil down to ...


1

The use of a database is great for helping you organize/find simulation data (Search by protein, search by simulation parameters). The database should then tell you where to find the relevant information on disk, where I imagine it is likely best stored on a per-simulation run basis in whatever file type is most convenient to load for analysis (whether ...


1

We have been using ZFP with HDF5 for our medical imaging data. It is made for lossy, floating point compression. We're running it on literally everything, and have more than 40TB of data stored (and being used!). It is fast enough to save our data real-time, and we can specify the required precision, so while the format is lossy, we're not seeing any ...


1

Possible methods, that can be used for floating-point compression: Transpose 4xN for float and 8xN for double + lz77 Implementation: Floating point compression in TurboTranspose see also error-bounded lossy compression Predictor (ex. Finite Context Method) + encoding (ex. "integer compression"). Implementation: Floating point compression in TurboPFor ...


1

SZ (developed by Argonne in 2016) could be a good choice. SZ: Fast Error-Bounded Floating-point Data Compressor for Scientific Applications https://collab.cels.anl.gov/display/ESR/SZ


Only top voted, non community-wiki answers of a minimum length are eligible