Tagged Questions

The study of collection, organization, analysis, and interpretation of data.

learn more… | top users | synonyms

0
votes
0answers
16 views

How to separate model errors from measurement errors? [on hold]

I've got data measured with errors that have known measured_RMS. I am testing a model that has model_RMS differences between model prediction and measured data. What would be a reasonable estimate of ...
0
votes
0answers
31 views

How to do Outlier Detection for data of an unknown function $f(x)$

Suppose I have an unknown function $f(x)$ that is monotonic, convex and nonlinear by definition. I somehow evaluated it at 1000 different $x$ points. By ploting these data, I found a very obvious ...
0
votes
2answers
108 views

Statistical analysis of optimization algorithms

If we optimize some parameter using 4 optimization algorithms, 2 of which are population based (say A and B) and 2 trajectory methods (single point search)(say C and D); what statistical test can be ...
0
votes
0answers
25 views

Monte Carlo test for spatio-temporal randomness

I am trying to establish statistical significance of discrete model observations $u_0(x,y,z,t)$ in relation to discrete observations $d(x,y,z,t)$ - both on the surface of a sphere. Typically both ...
2
votes
2answers
89 views

fastest and most efficent way to count all combinations in many sets and sum them together

I am a Java programmer who has reached the limits of brute computer power. My relational database (and non relational databases) is not producing results quick enough and I have hit a bottleneck in ...
1
vote
1answer
98 views

What is the algorithm that matlab used in its built-in function 'pca'?

Do anyone know what is the algorithm that MATLAB used in its built-in function "pca"? I have the following data set: 148.9820 55.8438 210.2150 149.3030 56.8891 208.4280 151.4400 ...
0
votes
2answers
42 views

What is 'SOLVER' in R and Statistics/Analytics ?

**strong text**a) I tried to research on what exactly is a SOLVER only to find a not clear-cut simple answer. My doubts still remain after going through several sites full of discussions about it. I ...
2
votes
2answers
75 views

Python: What is a good way to generate a 1D particle field with a gaussian distribution?

If I have N particles how do I assign their x values so that the end result is Gaussian distribution. i.e. particles near the ends are more spread out than particles near the center.
0
votes
0answers
6 views

Detect thermalization bias in a set of measures

I ran a simulation in which I initialised a system, gave it some time to thermalise, and then performed a few measurements. The measured values are the following: ...
0
votes
0answers
13 views

Model estimation using ACF and PACF

CAN ANYONE HELP IN MODEL ESTIMATION? The following are the ACF,PACF and the plot of the sample respectively.
0
votes
0answers
25 views

Mapping protein sequences on an abscissa?

I have asked this question in the stack forum but I think here, it's more appropriate. What would you do to represent the distribution of small sequences (<=40 residues) according to their ...
1
vote
1answer
147 views

Coding complex equations into C++

I have here equations from a paper by E. Bradlow. They're for counting the events in a Weibull-distributed data set. $$\begin{align} \Pr(N(t)=n) &= \sum_{j=n}^\infty{\frac{(-1)^{j+n}(\lambda ...
12
votes
3answers
113 views

Correct statistics for reporting speedup results

Say I have slow and fast versions of some code, and want to report a speedup number comparing the two. I run the slow version $n$ times and the fast version $m$ times, producing times $(s_1, \ldots, ...
1
vote
3answers
90 views

Quality of randomness in generated random number

I have generated a list of 3 random number where each summed to 1. I would like to assess the quality of randomness. What is the best mechanism to assess this randomness? E.g my random numbers are. ...
4
votes
1answer
63 views

Is it possible to compute quantiles of a set of numbers, without first sorting those numbers?

I apologize in advances if this is the wrong place to be asking this question, (I had considered putting it on CV first). I am studying financial (S&P, Dow Jones, etc) data, and would like to ...
0
votes
0answers
33 views

Statistical Coupling Analysis (SCA) to identify coevolved residues in protein families: use of ICA

This is a computational (structural) biology question. We've recently begun to try out the SCA Matlab toolbox (latest version) downloaded from Dr. Rama Ranganathan's website, and, following the ...
1
vote
1answer
55 views

Difference powerlaw, lognormal and streteched exponential (Weibull) function

I am currently fitting above mentioned functions to my data and I can observe, that both lognormal and Weibull are better fits than powerlaw. In literature, it is often suggested, that it is hard to ...
5
votes
1answer
142 views

Generation of variable with given auto-correlation function

How can I generate realizations of random complex variable x(t) with given auto-correlation function C(s), defined by C(s) = < x(s)x(0) > and obeying the condition C(-s) = C^*(s) ? Any link ...
1
vote
1answer
222 views

Information from residuals in polynomial regression

I have two physical properties which are exact (no noise). If I perform a polynomial regression of any degree, all coefficients are statistically significant. The problem is that the residuals are ...
1
vote
0answers
65 views

Differential equation - Model Fitting

My question is a quite simple one about statistics. I need to know the following thing. I have a differential equation from my model which I can solve. Now I can compare this solution with the ...
3
votes
0answers
112 views

How to estimate if a velocity field is statistically homogeneous?

For a 3D velocity field $\mathbf{u}$ obtained by direct numerical simulation. Assuming that the field is defined on a periodic domain $\mathcal{P}$ of periodicity $L_x$ in the $x$ direction, $L_y$ ...
1
vote
1answer
1k views

Autocorrelation problems

I have written some code, to produce some data. In what I have shown below, the output is just one-dimensional array of numbers. I have an analytical expression for what the autocorrelation function ...
2
votes
0answers
328 views

What is average pooling

I was reading an article where they have a set of pages. For every page they extract a feature vector of dimention 300. Then they fuse these vectors into one vector by using a method called average ...
6
votes
1answer
124 views

What computational methods would allow me to rank 2D surfaces (with examples)

I have a program which compares the similarity of two images for different positions, so my surface consists of points which correspond to X and Y translations each with a value (mutual information). ...
1
vote
1answer
57 views

Extract clusters from a graph of absolute-distance edge

I don't know if I have formulated this problem right: I have tons of items and the distances between each pair of them. Feeding this data into some visualization tool, I am able to create a nice ...
0
votes
0answers
298 views

matlab program to calculate Maxwell-Boltzmann distribution

I am trying run a Matlab program to number density of Maxwell-Boltzmann statistics (non relativistic limit) $n= \frac{m^2}{2}\pi^2 \exp(\frac{-1}{T})K2(\frac{m}{T})$ where $K2$ is the Bessel ...
7
votes
2answers
719 views

Is there a fast way to compute histograms for high-dimensional large datasets?

Currently the way I compute histograms for data is by generating grid in $N$ dimensions (where $N$ is the dimension of the data) and searching through the $M$ data points in each dimension to see in ...
4
votes
2answers
111 views

Hardware random number generator Vs. Pseudo random number generator in the battlefield of Markov Chain Monte Carlo processes

I'm implementing a Markov Chain Monte Carlo process for a Quantum Monte Carlo routine, in every book and paper I've read so far the success of the routine and quality of the results strongly depends ...
0
votes
0answers
70 views

How to avoid over fitting? An application of GBM boosting package

I distributed my data into 90% training and 10% testing, then I build a boosting regression model using GBM boosting package in R which is very similar to the famous Adaboost regression model. There ...
4
votes
1answer
557 views

TypeError from scipy.optimize.curve_fit

I am trying to fit a data set to an exponential model using scipy. However, the covariance matrix that is returned is 'inf' and I receive the following error: Traceback (most recent call last): ...
3
votes
1answer
42 views

How to obtain the minimum set of variables required in a model to produce accurate estimation?

I have a system which I assume is linear. I have a matrix $A$ of which each row is a coefficient of a unknown variables in vector $x$. I have vector $B$ which contains the result of each $Ax$. ...
-1
votes
1answer
79 views

quantifying interaction between variables in an equation

What do I need to measure interaction between variables in a particular equation? For e.g. Me just taking 50 grams of protein everyday will help me health wise. Me just doing exercise for 1 hour ...
2
votes
0answers
29 views

An asynchronous version of the Covariance Matrix Adaption (CMA)?

Does anyone know of a variant of the CMA algorithm that is suitable for an asynchronous parallel implementation? The conventional version of the algorithm allows one to evaluate the fitness function ...
10
votes
1answer
166 views

Statistical models for local memory/compute, network latency, and bandwidth jitter in HPC

Parallel computation is frequently modeled using a deterministic local rate of computation, latency overhead, and network bandwidth. In reality, these are spatially variable and non-deterministic. ...
1
vote
1answer
147 views

Clustering data based on the radius of a sphere

I have a data-set that has four columns [X Y Z C]. I would like to find all the C values that are in a given sphere centered at [X, Y, Z] with a radius r. What is the best approach to address this ...
2
votes
1answer
170 views

$k$-Nearest Neighbor Search using examples

I want to perform $k$-Nearest Neighbor Search in multidimensional space, but not using for example $L_2$-distance. I want the user to specify some "similar"-pairs examples and then perform a search ...
1
vote
1answer
107 views

Calculating AIC (in R or any other statistical software)

I am trying to fit the log-log plot of the cumulative distribution of a network to one of three models: Exponential (EXP) ($P(k)\sim e^{-ak}$), Exponentially truncated power law (TRU) ($P(k)\sim ...
3
votes
1answer
93 views

How to add perturbation to a base state in physical grid space?

For study 3D instabilities problem i.e a base state (velocity field and temperature field) and an additional perturbation. The method must be independent of the type of instabilities considered but ...
8
votes
3answers
204 views

How to estimate the impact of small scale on large scale in fluid dynamics?

Assuming that a direct numerical simulation is performed, what is a good method for estimate the impact of small scale on large scale in fluid dynamics ? For example is it pertinent to compare two run ...
2
votes
1answer
136 views

Root Convergence rate of Iterative Scheme

I have an iterative sequence for optimizing an EM (Expectation Maximization) algorithm based loss function $L(X)$ with $t$ being the iteration number as: $X_t=ABX_{t-1}+CX_{t-1}+X_{t-1}$ where $A$ is ...
2
votes
1answer
202 views

In molecular dynamics (MD) simulations, how is particle number density computed in practice?

I have been reading a recent paper. In it, the authors performed molecular dynamics (MD) simulations of parallel-plate supercapacitors, in which liquid resides between the parallel-plate electrodes. ...
5
votes
3answers
271 views

Computing the PDF of a quadratic function of two random variables

Given the function $\mathcal{M} = g + Ah + Bh^2$ where $A$ and $B$ are constants and $g$ and $h$ are random variables with their distributions $f_G(g)$ and $f_H(h)$ known, is it possible to compute ...
0
votes
0answers
114 views

Permissive Math Library for Parameter Statistics in C++

I am looking for recommendations for a C++ math library, with a permissive licence, well suited to calculating a wide variety of statistics on segmentations of timebased parameter data. I would be ...
5
votes
0answers
124 views

Fast algorithms to solve Markov Decision Processes

In my master thesis I used an Algorithm called Approximative Dynamic Programming [1] to solve equations of the form $$ ...
8
votes
2answers
2k views

Define custom probability density function in Python

Is there a way, using some stablished or known Python package (i.e. SciPy) to define my own probability density function (without any prior data, just $f(x) = a x + b$), so I can then make ...
5
votes
4answers
318 views

How can I reduce the error of the sample average?

I want to use the sample average $(X_1 + .... X_n)/n$ as a substitute for the expectation $\mathbb{E}(X)$. As claimed by the weak law of large numbers, as n increases the sample average should ...
3
votes
1answer
154 views

Computing a rolling quantile

An algorithm I'm writing needs to compute rolling quantiles of a time series. Currently I do this in the naive way: for a window of size W and a vector ...
3
votes
1answer
63 views

Adding deliberate imperfection to RNG output - toolkits?

Are there any existing software toolkits, libraries, frameworks or whatever for studying the quality of pseudorandom number generators that allow one to add a known amount of imperfection - e.g ...
16
votes
3answers
2k views

How to add large exponential terms reliably without overflow errors?

A very common problem in Markov Chain Monte Carlo involves computing probabilities that are sum of large exponential terms, $ e^{a_1} + e^{a_2} + ... $ where the components of $a$ can range from ...
3
votes
0answers
124 views

Probabilistic algorithms for matrix approximation

Considering regular matrix approximation inequality || $A - QQ^TA $|| < e where we try to approximate matrix $A$ by a lower rank orthonormal matrix $Q$. I've read an article on probabilistic ...