Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-storage]

The tag has no usage guidance.

12
votes
1answer
2k views

Alternatives to hdf5

I've been using HDF5 for years, but as the size of the dataset grows I'm starting to experience the same problems listed here http://cyrille.rossant.net/moving-away-hdf5/ Can you point me to a ...
6
votes
1answer
139 views

Storing large datasets on virtual filesystems

Reposted from: https://stackoverflow.com/q/34909093/472610 I work in scientific research. Technical tools aren't always ideal or even really thought through sometimes, but with recent developments ...
1
vote
1answer
864 views

Skyline solver for AX=B where A is symmetric skyline matrix

I am looking for a simple subroutine in Fortran 90 (GNU Compiler) to solve linear equation of the type $AX=B$, where $A$ is an $n\times n$ symmetric matrix stored in the form of symmetric skyline ...
4
votes
1answer
2k views

Eigen - store sparse matrix as binary

I need to store large sparse matrices in Eigen. I cannot find anything in the library except the function below, in Eigen/Unsupported. The problem with saveMarket is, that it saves in text format. Due ...
2
votes
1answer
122 views

Topics about the deal.II finite element library class “SparsityPattern”

When learning the deal.II FE library, I am a bit confused about the mechanism of its "SparsityPattern" class. Through reading the documentation, I only got to know that it uses the Compressed Row ...
1
vote
1answer
64 views

Maple stored variable to be read into Matlab

I have a a function $f(k)$ (calculated in Maple) which is huge and stored in a variable called 'sum' on my drive (with the help of 'save' command in Maple). Since the function is huge, Maple is unable ...
2
votes
2answers
1k views

How best to use scipy.integrate's ODE solvers when state is not naturally stored as a vector?

I have a large system of ODEs. For various reasons, it is natural to store the values of the dependent variables in a multidimensional array. For example, these values might represent the solution ...
3
votes
0answers
66 views

What disables Gradual Overflow?

After studying the process of Gradual Underflow, I'm left a little curious as to why machines don't implement Gradual Overflow; where numbers exceeding the overflow level would be stored as un-...
11
votes
3answers
1k views

Are HDF5 files suitable for git revision control?

I am not familiar with the file format used in HDF5, but I am wondering if HDF5 files are suitable for revision control with git (or for example Mercurial or Subversion)? I guess what I mean is: are ...
5
votes
3answers
3k views

Storing a large, sparse array for R and Python

I've been working in R but sometimes switching to python. I'd like a more inter-language portable way of storing a large array than a csv file. (The particular csv file I'm dealing with is about 10^6 ...
13
votes
3answers
1k views

Best practice for storing hierarchical simulation data

TL,DR What is the accepted best practice in scientific computing circles for storing large quantities of hierarchically structured data? For example, SQL does not play nicely with large sparse ...
26
votes
9answers
6k views

Compressing floating point data

Are there any tools specifically designed for compressing floating point scientific data? If a function is smooth, there's obviously a lot of correlation between the numbers representing that ...
5
votes
1answer
301 views

Working with multi-dimensional functions

How would you represent functions of type $[-1, 1]^n \to \mathbb R \;$ for moderate $n$? How would you integrate them? For small $n$ (1-2) such functions can be represented as histograms, vectors in ...