Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [data-management]

The tag has no usage guidance.

3
votes
3answers
130 views

Simple way to store/read data from file in C++

I've been running various simulations with C++, and doing so has often involved saving lots of data to file (real/complex matrices, arrays, etc) and then reading them into other programs later. ...
1
vote
3answers
190 views

Best way to track parameters and results, in your experience (i.e. how do you do it)?

Computational Science has many more parameters than my previous programming, and I'm finding it difficult to keep track. What's the best way to track them? editing source, commenting-out. Good for ...
4
votes
0answers
579 views

Neural network performs worse when using more input variables

This question is based more on the theory of neural networks than my particular implementation. Therefore I will leave out my code unless requested. I'm working on a project in C# which can create ...
0
votes
1answer
492 views

Importing results of FEM analysis into Matlab

I need to import in Matlab the results (like time histories of diplacement or frequency response at a specific point) obtained from a FEM analysis in Nastran. At the moment I ask Nastran to save the ...
4
votes
7answers
450 views

Recommendations for input file format for science applications

Assuming that a program is written in either C, C++, Fortran or python, what do you think is the best format for the input file given that users could be relatively diverse, i.e. not familiar with the ...
3
votes
2answers
279 views

Cloud solutions for microscopy image analysis

At our research facility we routinely perform what are called PALM (Photo-activated localisation microscopy) or STORM (stochastic optical reconstruction microscopy) experiments. You can read all ...
10
votes
2answers
10k views

Pandas limitations and its alternatives in Python

I read somewhere that Pandas was first developed for the financial world, at least not especially for natural sciences (physics, biology and so on), so is there any similar data analysis Python ...
1
vote
1answer
57 views

Searching in a large dataset

I am working on a model of social interactions in mice. I have mice and boxes and a simulation that outputs which mouse stays in which box during which time period (this comes out as a mouse is ...
9
votes
2answers
292 views

suggestion for managing simulation runs?

This questions may be a bit off-topic in comp-sci. if it is needed please suggest where does it fit with. The question is regarding on how to manage all the simulation runs efficiently. let's say, ...
11
votes
3answers
1k views

Are HDF5 files suitable for git revision control?

I am not familiar with the file format used in HDF5, but I am wondering if HDF5 files are suitable for revision control with git (or for example Mercurial or Subversion)? I guess what I mean is: are ...
7
votes
2answers
3k views

Is there a fast way to compute histograms for high-dimensional large datasets?

Currently the way I compute histograms for data is by generating grid in $N$ dimensions (where $N$ is the dimension of the data) and searching through the $M$ data points in each dimension to see in ...
0
votes
1answer
85 views

mapping data with a spike to a heat map

I have the following dataset that I need to display on the heat map: [30, 15, 66, 7, 9999, 78, 42, 132] So if I map the values to the color scale using a linear function I only see the spike while ...
0
votes
3answers
259 views

Translating from SBML to Petri Net (matrix representation)

Is there a piece of software that can easily take a representation of a biochemical network from a SBML file and translate it into a Petri Net? Specifically, I'm looking for something that can give ...
2
votes
3answers
8k views

Dealing with non-monotonically increasing data

How do you deal with data that you need to be monotonically increasing in order to work with interpolation libs and other functions, when it is in fact not monotonically increasing?
5
votes
3answers
3k views

Storing a large, sparse array for R and Python

I've been working in R but sometimes switching to python. I'd like a more inter-language portable way of storing a large array than a csv file. (The particular csv file I'm dealing with is about 10^6 ...
13
votes
3answers
1k views

Best practice for storing hierarchical simulation data

TL,DR What is the accepted best practice in scientific computing circles for storing large quantities of hierarchically structured data? For example, SQL does not play nicely with large sparse ...
4
votes
2answers
271 views

Memory usage within graphics cards

I have a question about the memory usage within the graphics card. What does happen when the used application needs more memory than the graphic card does contain? let`s say the reserved main memory ...
5
votes
1answer
69 views

Are there any open standards for data exchange between components of a spatial model?

I'm just getting into climate research, and am amazed at how much poorly documented legacy code is used. This kind of thing means that models more or less have to be maintained by the people who wrote ...
40
votes
14answers
18k views

Which software and workflow is recommend for publication of scientific data and graphs?

Which software provides a good workflow from simple plotting of a few datapoints up to the creation of publication level graphics with detailed styles, mathematical typesetting and "professional ...
6
votes
2answers
375 views

Filtering a dataset to get a more uniform distribution for neural network training

I'm looking into using artificial neural networks (ANN) to predict the reaction rates in my fluid instead of solving the full system of stiff ODEs. Some people from my lab have already done some work ...
4
votes
3answers
285 views

How to fill a 2D set over a cartesian lattice with as few rectangles as possible?

Suppose I have a black and white image (composed of binary pixel values in a 2D cartesian array) that contains an irregular, nonconvex shape. Let's further suppose that the shape is one connected ...
2
votes
3answers
908 views

List of data munging libraries

I am trying to put together a complete list of all of the tools that computational scientists have found useful when trying to munge data, i.e. take data in one format, extract the useful bits, and ...
14
votes
4answers
345 views

How to deal with too much data?

Our plasma dynamics simulations often produce too much information. During the simulations we record various physical properties on a grid (x,y,z,t) that is as large as (8192x1024x1024x1500), for at ...