The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
0answers
32 views

How to do Outlier Detection for data of an unknown function $f(x)$

Suppose I have an unknown function $f(x)$ that is monotonic, convex and nonlinear by definition. I somehow evaluated it at 1000 different $x$ points. By ploting these data, I found a very obvious ...
0
votes
2answers
43 views

What is 'SOLVER' in R and Statistics/Analytics ?

**strong text**a) I tried to research on what exactly is a SOLVER only to find a not clear-cut simple answer. My doubts still remain after going through several sites full of discussions about it. I ...
5
votes
1answer
109 views

Dimensionality reduction of the domain of f(x)

I'm wondering if there is something analogous to a PCA for data sets where there is a dependent variable. (Though I am interested in any method of dimensionality reduction, PCA is just an example.) ...
0
votes
0answers
16 views

Finding correlations between sparse data with different temporal bases

The problem was one that came up during a standards committee meeting. If have one channel of data sampled every 100ms to record fault codes on a system (say an gasoline engine). It records value ...
3
votes
1answer
82 views

Monte Carlo Simulation - Random Number Motivation

For Monte Carlo simulations, or any other numerical methods that rely heavily on the quality of the pseudo-random numbers generated (i.e even/desired distribution on a certain domain) for that matter, ...
5
votes
4answers
171 views

Should I use a database to handle large amounts of results?

Background: I am currently running a large amount parameter variation experiments. They are being run in Python 2.6+, using numpy. These experiments are going to take about 2 weeks to run. Roughly I ...
0
votes
0answers
6 views

Detect thermalization bias in a set of measures

I ran a simulation in which I initialised a system, gave it some time to thermalise, and then performed a few measurements. The measured values are the following: ...
1
vote
0answers
70 views

SciDB vs OLAP cube

I am working on good retrieval mechanism for big data systems. I am of view that multidimensional structures could be of great use for aggregations and querying . Multidimensional structures as ...
0
votes
0answers
36 views

how to detect planets in resonance

I make this program, i have a 2xN matrix in which the columns are the ID of planets and their period, the rows are the number of planets, for istance something like that: ...
2
votes
2answers
72 views

Extracting time scales information from empirical cumulative distribution function

I have a stochastic process (a Markov chain actually) that has two absorbing states. I am using a difference equation to calculate the first passage time to either of the absorbing states. There are ...
0
votes
1answer
47 views

Finding maximum value in huge dataset

I am working on a project involving some pattern recognition. For this I need to find the maximum value in a huge multidimensional dataset. For example I have a discrete 5-dimensional space containing ...
2
votes
0answers
45 views

Spatio-temopral wavelet analysis

Am quite new to wavelet analysis and would like some help. I am performing a spatio-temporal analysis of monthly gridded rainfall data. With PCA, I can reduce the dimension of the rainfall data into a ...
5
votes
2answers
436 views

Pandas limitations and its alternatives in Python

I read somewhere that Pandas was first developed for the financial world, at least not especially for natural sciences (physics, biology and so on), so is there any similar data analysis Python ...
2
votes
3answers
146 views

R/C/C++ library for N-dimensional arrays

I'm looking for a library that is either in R or easily wrappable with R, that can do the following things: construct and subset N-dimensional arrays perform operations such as ...
5
votes
1answer
218 views

Fit my data to Lissajous curve in Matlab

I'm acquiring data which looks like this: Theoretically it should be possible to fit it with a Lissajous curve, which is typically defined in the following way: $x = A \sin(a t + \delta)$, and $y = ...
1
vote
1answer
34 views

Line scan in a 2D histogram

So I have a 2D histogram, in the third dimention are the counts. What I want to achieve is by choosing two random position, make a plot of a 1D histogram along the line that passes through the two ...
1
vote
0answers
33 views

Spectra of the energy flux VS transfer spectra?

In direct numerical simulation of turbulence what is the advantage to look at the spectra of the energy flux rather than the transfer spectra? The transfer spectra corresponds to the triple-velocity ...
1
vote
1answer
32 views

Adding data with different abscissas

This question may be better suited for an Astronomy Stack Exchange site, but I figured I'd ask here. Say I have measurements of something as a function of radius for a number of objects. Here's an ...
3
votes
2answers
56 views

Parameter Fitting: Need measure of data 'support' for a parameter solution

I am estimating parameters on a dataset that would, for the most part, result in a weakly constrained solution. The dataset however also contains a few more data points that make the solution ...
1
vote
1answer
57 views

Extract clusters from a graph of absolute-distance edge

I don't know if I have formulated this problem right: I have tons of items and the distances between each pair of them. Feeding this data into some visualization tool, I am able to create a nice ...
2
votes
0answers
47 views

Error in treating non-uniformly spaced data as uniform

In discussing smoothing filters, Numerical Recipes p. 772 says ... irregularly sampled data, where the values $f_i$ are not uniformly spaced ... one can simply pretend that the data points are ...
3
votes
1answer
323 views

How to do local FFT on huge 3D vector data cell mesh and visualize it spatially?

Simulation type: I'm running a simulation with the OOMMF micromagnetics package http://math.nist.gov/oommf/ where are magnet is represented by a mesh of 3 million cells, it gets excited by a ...
1
vote
1answer
89 views

classification machinery needed

Consider a set of 7D vectors. Each vector belongs to one of four classes. After mapping to 3D with PCA and coloring each point according its class the dataset looks like as shown below: For the ...
1
vote
1answer
153 views

Clustering data based on the radius of a sphere

I have a data-set that has four columns [X Y Z C]. I would like to find all the C values that are in a given sphere centered at [X, Y, Z] with a radius r. What is the best approach to address this ...
5
votes
1answer
206 views

Recommendation for a python book for data processing

I've been doing FORTRAN programming for 10 years and I've started using python for a few years now, mostly for data processing. I've been lucky enough to work with people that are knowledgeable ...
1
vote
2answers
63 views

Estimate (non-)drift in noisy data

I have a time series representing the result of a complex calculation (physical simulation). Due to round-off errors and approximation errors, there will be some "noise" on the data series. In some ...
9
votes
4answers
3k views

Fastest PCA algorithm for high-dimensional data

I would like to perform a PCA on a dataset composed of approximately 40 000 samples, each sample displaying about 10 000 features. Using Matlab princomp function consistently takes over half an hour ...
3
votes
1answer
83 views

Calculation of Multivariate Coherence

I trying to detecting whether a data set of time series has a global change in frequencies. Calculating the average (or median) pairwise coherence, I feel, misses the point because I am trying to get ...
2
votes
0answers
638 views

Can post-processing use topoSet and createPatch without screwing up the results?

I would like to use patchAverage to obtain the average pressure on an object that consists of multiple patches (due to different boundary conditions), however ...
9
votes
2answers
428 views

Best practice for storing hierarchical simulation data

TL,DR What is the accepted best practice in scientific computing circles for storing large quantities of hierarchically structured data? For example, SQL does not play nicely with large sparse ...
10
votes
2answers
102 views

Matching Similar Items from a Set

I'm trying to match items. Given a set of $n$ items I can rank on a scale from 0 to 100 of how similar they are to one another. For instance, if item $n_1$ is milk and item $n_2$ is also milk, then ...