Questions tagged [data-management]

For questions about how best to process and store the data produced from calculations.

Filter by
Sorted by
Tagged with
1 vote
2 answers
3k views

Robust alternative to HDF5 for storing large datasets

When running my simulations I usually get result files with typical sizes of 10 GByte and larger. They contain vectors and matrices with data type float, double, complex float and complex double. ...
arc_lupus's user avatar
  • 543
1 vote
1 answer
120 views

k nearest neighbors dictionary for vectors?

Does there exists a data structure with these properties Stores vectors in a dictionary When given a key that is also a vector, returns the k nearest vectors using a similarity metric (e.g. Euclidean,...
user8714896's user avatar
4 votes
3 answers
7k views

Simple way to store/read data from file in C++

I've been running various simulations with C++, and doing so has often involved saving lots of data to file (real/complex matrices, arrays, etc) and then reading them into other programs later. ...
Henry Shackleton's user avatar
1 vote
3 answers
232 views

Best way to track parameters and results, in your experience (i.e. how do you do it)?

Computational Science has many more parameters than my previous programming, and I'm finding it difficult to keep track. What's the best way to track them? editing source, commenting-out. Good for ...
hyperpallium's user avatar
4 votes
0 answers
1k views

Neural network performs worse when using more input variables

This question is based more on the theory of neural networks than my particular implementation. Therefore I will leave out my code unless requested. I'm working on a project in C# which can create ...
F Chopin's user avatar
  • 141
0 votes
1 answer
730 views

Importing results of FEM analysis into Matlab

I need to import in Matlab the results (like time histories of diplacement or frequency response at a specific point) obtained from a FEM analysis in Nastran. At the moment I ask Nastran to save the ...
Rhei's user avatar
  • 402
5 votes
7 answers
2k views

Recommendations for input file format for science applications

Assuming that a program is written in either C, C++, Fortran or python, what do you think is the best format for the input file given that users could be relatively diverse, i.e. not familiar with the ...
FrenchKheldar's user avatar
3 votes
2 answers
407 views

Cloud solutions for microscopy image analysis

At our research facility we routinely perform what are called PALM (Photo-activated localisation microscopy) or STORM (stochastic optical reconstruction microscopy) experiments. You can read all ...
Kris's user avatar
  • 131
11 votes
2 answers
11k views

Pandas limitations and its alternatives in Python

I read somewhere that Pandas was first developed for the financial world, at least not especially for natural sciences (physics, biology and so on), so is there any similar data analysis Python ...
Clark's user avatar
  • 211
1 vote
1 answer
63 views

Searching in a large dataset

I am working on a model of social interactions in mice. I have mice and boxes and a simulation that outputs which mouse stays in which box during which time period (this comes out as a mouse is ...
user3000568's user avatar
9 votes
2 answers
443 views

suggestion for managing simulation runs?

This questions may be a bit off-topic in comp-sci. if it is needed please suggest where does it fit with. The question is regarding on how to manage all the simulation runs efficiently. let's say, ...
Chenming Zhang's user avatar
14 votes
3 answers
2k views

Are HDF5 files suitable for git revision control?

I am not familiar with the file format used in HDF5, but I am wondering if HDF5 files are suitable for revision control with git (or for example Mercurial or Subversion)? I guess what I mean is: are ...
Thomas Arildsen's user avatar
7 votes
2 answers
4k views

Is there a fast way to compute histograms for high-dimensional large datasets?

Currently the way I compute histograms for data is by generating grid in $N$ dimensions (where $N$ is the dimension of the data) and searching through the $M$ data points in each dimension to see in ...
Ron's user avatar
  • 265
0 votes
1 answer
118 views

mapping data with a spike to a heat map

I have the following dataset that I need to display on the heat map: [30, 15, 66, 7, 9999, 78, 42, 132] So if I map the values to the color scale using a linear function I only see the spike while ...
Trident D'Gao's user avatar
2 votes
3 answers
391 views

Translating from SBML to Petri Net (matrix representation)

Is there a piece of software that can easily take a representation of a biochemical network from a SBML file and translate it into a Petri Net? Specifically, I'm looking for something that can give ...
tel's user avatar
  • 1,380
1 vote
3 answers
12k views

Dealing with non-monotonically increasing data

How do you deal with data that you need to be monotonically increasing in order to work with interpolation libs and other functions, when it is in fact not monotonically increasing?
pyCthon's user avatar
  • 121
5 votes
3 answers
5k views

Storing a large, sparse array for R and Python

I've been working in R but sometimes switching to python. I'd like a more inter-language portable way of storing a large array than a csv file. (The particular csv file I'm dealing with is about 10^6 ...
cjordan1's user avatar
  • 805
16 votes
2 answers
2k views

Best practice for storing hierarchical simulation data

TL,DR What is the accepted best practice in scientific computing circles for storing large quantities of hierarchically structured data? For example, SQL does not play nicely with large sparse ...
tel's user avatar
  • 1,380
4 votes
2 answers
286 views

Memory usage within graphics cards

I have a question about the memory usage within the graphics card. What does happen when the used application needs more memory than the graphic card does contain? let`s say the reserved main memory ...
K.K's user avatar
  • 41
5 votes
1 answer
72 views

Are there any open standards for data exchange between components of a spatial model?

I'm just getting into climate research, and am amazed at how much poorly documented legacy code is used. This kind of thing means that models more or less have to be maintained by the people who wrote ...
naught101's user avatar
  • 373
44 votes
14 answers
21k views

Which software and workflow is recommend for publication of scientific data and graphs?

Which software provides a good workflow from simple plotting of a few datapoints up to the creation of publication level graphics with detailed styles, mathematical typesetting and "professional ...
Alexander's user avatar
  • 541
8 votes
2 answers
487 views

Filtering a dataset to get a more uniform distribution for neural network training

I'm looking into using artificial neural networks (ANN) to predict the reaction rates in my fluid instead of solving the full system of stiff ODEs. Some people from my lab have already done some work ...
FrenchKheldar's user avatar
5 votes
3 answers
331 views

How to fill a 2D set over a cartesian lattice with as few rectangles as possible?

Suppose I have a black and white image (composed of binary pixel values in a 2D cartesian array) that contains an irregular, nonconvex shape. Let's further suppose that the shape is one connected ...
Paul's user avatar
  • 12k
2 votes
3 answers
953 views

List of data munging libraries

I am trying to put together a complete list of all of the tools that computational scientists have found useful when trying to munge data, i.e. take data in one format, extract the useful bits, and ...
tel's user avatar
  • 1,380
15 votes
4 answers
408 views

How to deal with too much data?

Our plasma dynamics simulations often produce too much information. During the simulations we record various physical properties on a grid (x,y,z,t) that is as large as (8192x1024x1024x1500), for at ...
Yann's user avatar
  • 852