9

You're going to get a much better answer if you provide a few more technical details about what kind of data you're trying to put under version control, how you want to store different versions of the data, what components are likely to change and what components aren't, and whether you're truly going to have tree-like history (branches, merges). HDF5 files ...


8

I would perhaps argue that the main virtue of XML is the ease with which one can write a parser, rather than the ease with which you can write an actual XML document. JSON seems a slightly better alternative to my mind. Both have the advantage of being standards to some degree, meaning people won't have to learn whichever arcane syntax your program of choice ...


6

Histograms are not useful for high dimensional data. The curse of dimensionality affects one quite fast. As in your case if the grid is of size 7**6, you have on average one point in one bin. Kernel density estimator are better suited as long as you keep the kernel bandwidth large enough. In my experience the top hat kernel as k-nearest neighbor yields ...


6

If you want to remove the nonmonotonic structure without changing anything else, you can do an isotonic least squares fit: http://en.wikipedia.org/wiki/Isotonic_regression


5

I agree with Davidmh that calculating uncertainties should not be handled by an automatic library. You will very quickly run into a case where the automatics fail (try doing a Fourier transform for instance). You say however that you just want to keep the uncertainties with your data. Why not just add them as an extra column in your dataframe? This is how I ...


5

TLDR Use Python to manage/modify your input and coral your output, and use HDF5 to organize/store your data. As complex as it might seem at first it'll still be simpler than SQL-anything. Longer answer + Example I personally use a combination of Python scripting and the HDF5 file format to deal with these kinds of situations. Python scripting can handle the ...


5

Numpy has a file format that is pretty simple, which makes it perfectly compatible with basically every other high level language. (https://www.numpy.org/devdocs/reference/generated/numpy.lib.format.html) It looks like the format is much lighter than boost or hdf5. The docs say it should be easy enough to write a parser yourself, if necessary, which I would ...


4

If you are interested in a "standardized" format, you should have a look at the matrix market exchange formats. The "coordinate format" (which is good for sparse matrices) simply adds metadata to the format suggested by Aron in his answer and specifies how the data has to be formatted.


3

I disagree with the "make it as easy as possible" answers to this question. Options 1 and 2 lock you into something you will have a hard time changing later on. For example, you will find that if your number of parameters grows, you will want to put them into sections and subsections because that makes it simpler to organize. But your syntax doesn't allow ...


3

Awesome to find another microscopist on this site. Welcome! Short Answer: There is no magic-bullet on the market that will solve this problem space for you. Longer Answer: Putting together a few commercailly available offerings gets you pretty far. At my work we take ~1-10 thousand $4\,096\times25\,000$ images weekly, which puts us in a similar ...


3

I guess what I mean is: are HDF5 files suitable for line-based diff'ing or will git have to treat an HDF5 as one big binary and store an entire copy for each revision? The literal answer to this question is that git will not treat HDF5 files efficiently. For more useful answers about version control for projects that have some binary files, see this ...


3

As others said, it would be easier to make useful suggestions if you described your overall goal rather than a precise technical point. Here is one more suggestion that might help you, depending on what your goal is. The ActivePapers project (http://www.activepapers.org/) provides a code and data management system on top of HDF5. An ActivePaper is an HDF5 ...


3

I'm surprised nobody has mentioned Nico Schlömer's excellent tools matlab2tikz and matplotlib2tikz yet. If you are using LaTeX for document preparation and either Matlab or Python for data processing, you can easily get high quality vector plots which you can post process to your heart's content: Prepare your plots in Matlab or Python, including axes, ...


3

My quick vote is for QtiPlot. Though it isn't perfect, it provides the best combination of GUI easiness, along with python scripting so what you're really doing then is "Qtiplot/Python/(Illustrator or Inkscape)". Qtiplot is extensible, so you can create your own scripts/macros which can run python commands to open and process data and then dump them into ...


3

Since you've asked about experiences, I've seen all three of the methods you suggest used. However best practise is that inputs to the code, and the code framework itself are kept separate. This means that only parameters you don't expect to ever need to change should be hard coded into your source. For example, if you were modelling the flight of ...


2

Assuming your sparse array is 2-dimensional, you can decompose it into three vectors of column (index), row (index), and value fairly easily with a single traversal of the matrix. You can then store these vectors in whatever file format you want, no need to switch to HDF5 for that reason alone.


2

There are many different ways of handling outliers, as well as answers to similar questions here already. I would recommend taking a look at https://stackoverflow.com/questions/9811866/ggplot2-color-scale-over-affected-by-outliers http://en.wikipedia.org/wiki/Outlier#Working_with_outliers The first deals with this problem in R. Taking the approach of ...


2

I think we'd need to know a little bit more about your workflow to make any serious recommendations. I'd suggest treating your runs like a key-value store. Create a simple database for all of your metadata for each run, and then hash any relevant information from your run into a key that you assign to each output. In the simplest situation, you would ...


2

I happened to have this little Python function kicking around: def storeSparseProblem(sp_mat, rhs, filename): # Convert the sparse matrix to lil sp_mat = sp_mat.tocoo() # Make a record array out of the above rcv = recarray((3, len(sp_mat.row)), dtype=[("row","<i8"), ("col","<i8"), ("val","<f8")]) # Set the values rcv["row"]...


2

I don't know what you can do to keep track of parameters and results, though @origimbo correctly noted that repeatability kinda makes results useless to store. The first point you make is obviously the versioning problem, so the solution is trivial: create a repository on Github or Bitbucket or Mercurial. These services are very effective and they let you ...


2

Managing uncertainties is actually a quite delicate statistics problem. The known expression for error propagation using squared partial derivatives is good when the errors are normally distributed, independent, and small. This is usually the case; and in fact, even if the normality or independence are not fully satisfied, for most practical cases the result ...


2

You need to think "out of the box" (pun intended). Because you have a database of stays in the boxes, you think of your algorithm of looping over all stays. But to find overlaps, you need to compare each stay with each other stay -- so your algorithm has ${\cal O}(N_s^2)$ complexity where $N_s$ is the number of stays. This is expensive, as you have noticed. ...


2

You can use boost's serialization facilities, which can be straightforwardly extended to support custom data structures. If you use Eigen you can adapt something like this in order to suit your needs.


2

Cereal is a straight-forward and easy-to-use serialization library for C++ data structures. It knows all of the STL data structures and adapting custom classes for use with it is easy. Example code: #include <cereal/types/unordered_map.hpp> #include <cereal/types/memory.hpp> #include <cereal/archives/binary.hpp> #include <fstream> ...


1

Use the dumbest format that suits your needs. Make it ASCII-based if you can. Make it inflexible and as easy to parse as possible because there are better things to do with your life than writing input parsers. That means eschewing JSON and XML and S-expressions and any sort of hierarchy that the code itself will have to strip away. Look at the DIMACS ...


1

This answer is like LKlevin's but too long to add as a comment. If it's OK for input file to not allow fully general (turing complete?) instructions then I'd use json. If you want to allow the user to specify turing complete instructions in a setting that is tightly controlled by your software then I'd use lua. If you want to let the user do anything in ...


1

Don't ignore HDF5 which is designed for scientific datasets and has bindings for all the languages you mention, more besides. It's widely used and there are tools for poking around inside files without programming. If you want to (though I don't know why you would) you can even dump an HDF5 file into XML. It's primary purpose may be for large datasets but ...


1

I can recommend based on my personal experience. I had to do a similar task of parsing an input file in a cross-platform C++ program. I needed something simple and most of all something that could be swiftly implemented, so I went with choice #1 (ASCII), because the input data was relatively plain and simple. However, i think that the size and complexity ...


1

Based on our discussion above, I'd recommend the following: Find your local, regional, or national supercomputing, HPC, or research computing outfit and learn about their support for these kinds of data storage and computing activities. This kind of workload is one of the many things covered under the umbrella of "Big Data", and all of the supercomputing ...


Only top voted, non community-wiki answers of a minimum length are eligible