The data-management tag has no wiki summary.
6
votes
2answers
69 views
Is there a fast way to compute histograms for high-dimensional large datasets?
Currently the way I compute histograms for data is by generating grid in $N$ dimensions (where $N$ is the dimension of the data) and searching through the $M$ data points in each dimension to see in ...
0
votes
0answers
67 views
Tecplot: Overlapping of 2 contour plot data from a single file, in a 2D Cartesian grid
I have a rectangular grid ASCII .dat file that looks like this
...
0
votes
1answer
25 views
mapping data with a spike to a heat map
I have the following dataset that I need to display on the heat map:
[30, 15, 66, 7, 9999, 78, 42, 132]
So if I map the values to the color scale using a linear function I only see the spike while ...
0
votes
3answers
64 views
Translating from SBML to Petri Net (matrix representation)
Is there a piece of software that can easily take a representation of a biochemical network from a SBML file and translate it into a Petri Net?
Specifically, I'm looking for something that can give ...
0
votes
3answers
501 views
Dealing with non-monotonically increasing data
How do you deal with data that you need to be monotonically increasing in order to work with interpolation libs and other functions, when it is in fact not monotonically increasing?
4
votes
3answers
479 views
Storing a large, sparse array for R and Python
I've been working in R but sometimes switching to python. I'd like a more inter-language portable way of storing a large array than a csv file. (The particular csv file I'm dealing with is about 10^6 ...
9
votes
2answers
254 views
Best practice for storing hierarchical simulation data
TL,DR
What is the accepted best practice in scientific computing circles for storing large quantities of hierarchically structured data? For example, SQL does not play nicely with large sparse ...
4
votes
2answers
178 views
Memory usage within graphics cards
I have a question about the memory usage within the graphics card. What does happen when the used application needs more memory than the graphic card does contain? let`s say the reserved main memory ...
5
votes
1answer
58 views
Are there any open standards for data exchange between components of a spatial model?
I'm just getting into climate research, and am amazed at how much poorly documented legacy code is used. This kind of thing means that models more or less have to be maintained by the people who wrote ...
19
votes
11answers
2k views
Which software and workflow is recommend for publication of scientific data and graphs?
Which software provides a good workflow from simple plotting of a few datapoints up to the creation of publication level graphics with detailed styles, mathematical typesetting and "professional ...
5
votes
2answers
238 views
Filtering a dataset to get a more uniform distribution for neural network training
I'm looking into using artificial neural networks (ANN) to predict the reaction rates in my fluid instead of solving the full system of stiff ODEs. Some people from my lab have already done some work ...
4
votes
3answers
201 views
How to fill a 2D set over a cartesian lattice with as few rectangles as possible?
Suppose I have a black and white image (composed of binary pixel values in a 2D cartesian array) that contains an irregular, nonconvex shape. Let's further suppose that the shape is one connected ...
1
vote
3answers
271 views
List of data munging libraries
I am trying to put together a complete list of all of the tools that computational scientists have found useful when trying to munge data, i.e. take data in one format, extract the useful bits, and ...
12
votes
4answers
265 views
How to deal with too much data?
Our plasma dynamics simulations often produce too much information. During the simulations we record various physical properties on a grid (x,y,z,t) that is as large as (8192x1024x1024x1500), for at ...
