Questions tagged [statistics]

The study of collection, organization, analysis, and interpretation of data.

Filter by
Sorted by
Tagged with
2
votes
2answers
113 views

fastest and most efficent way to count all combinations in many sets and sum them together

I am a Java programmer who has reached the limits of brute computer power. My relational database (and non relational databases) is not producing results quick enough and I have hit a bottleneck in ...
1
vote
1answer
217 views

What is the algorithm that matlab used in its built-in function 'pca'?

Do anyone know what is the algorithm that MATLAB used in its built-in function "pca"? I have the following data set: 148.9820 55.8438 210.2150 149.3030 56.8891 208.4280 151.4400 55....
0
votes
2answers
853 views

What is 'SOLVER' in R and Statistics/Analytics ?

**strong text**a) I tried to research on what exactly is a SOLVER only to find a not clear-cut simple answer. My doubts still remain after going through several sites full of discussions about it. I ...
2
votes
2answers
698 views

Python: What is a good way to generate a 1D particle field with a gaussian distribution?

If I have N particles how do I assign their x values so that the end result is Gaussian distribution. i.e. particles near the ends are more spread out than particles near the center.
1
vote
1answer
1k views

Coding complex equations into C++

I have here equations from a paper by E. Bradlow. They're for counting the events in a Weibull-distributed data set. $$\begin{align} \Pr(N(t)=n) &= \sum_{j=n}^\infty{\frac{(-1)^{j+n}(\lambda t^c)^...
12
votes
3answers
221 views

Correct statistics for reporting speedup results

Say I have slow and fast versions of some code, and want to report a speedup number comparing the two. I run the slow version $n$ times and the fast version $m$ times, producing times $(s_1, \ldots, ...
1
vote
3answers
193 views

Quality of randomness in generated random number

I have generated a list of 3 random number where each summed to 1. I would like to assess the quality of randomness. What is the best mechanism to assess this randomness? E.g my random numbers are. ...
4
votes
1answer
189 views

Is it possible to compute quantiles of a set of numbers, without first sorting those numbers?

I apologize in advances if this is the wrong place to be asking this question, (I had considered putting it on CV first). I am studying financial (S&P, Dow Jones, etc) data, and would like to ...
3
votes
1answer
344 views

Difference powerlaw, lognormal and streteched exponential (Weibull) function

I am currently fitting above mentioned functions to my data and I can observe, that both lognormal and Weibull are better fits than powerlaw. In literature, it is often suggested, that it is hard to ...
7
votes
1answer
187 views

Generation of variable with given auto-correlation function

How can I generate realizations of random complex variable $x(t)$ with a given autocorrelation function $C(s)$, defined by $$C(s) = \langle x(s) x(0) \rangle$$ and obeying the condition $C(-s) = C^*(...
2
votes
1answer
3k views

Information from residuals in polynomial regression

I have two physical properties which are exact (no noise). If I perform a polynomial regression of any degree, all coefficients are statistically significant. The problem is that the residuals are ...
3
votes
0answers
216 views

How to estimate if a velocity field is statistically homogeneous?

For a 3D velocity field $\mathbf{u}$ obtained by direct numerical simulation. Assuming that the field is defined on a periodic domain $\mathcal{P}$ of periodicity $L_x$ in the $x$ direction, $L_y$ ...
1
vote
1answer
3k views

Autocorrelation problems

I have written some code, to produce some data. In what I have shown below, the output is just one-dimensional array of numbers. I have an analytical expression for what the autocorrelation function ...
2
votes
0answers
1k views

What is average pooling

I was reading an article where they have a set of pages. For every page they extract a feature vector of dimention 300. Then they fuse these vectors into one vector by using a method called average ...
6
votes
1answer
213 views

What computational methods would allow me to rank 2D surfaces (with examples)

I have a program which compares the similarity of two images for different positions, so my surface consists of points which correspond to X and Y translations each with a value (mutual information). ...
1
vote
1answer
72 views

Extract clusters from a graph of absolute-distance edge

I don't know if I have formulated this problem right: I have tons of items and the distances between each pair of them. Feeding this data into some visualization tool, I am able to create a nice ...
7
votes
2answers
4k views

Is there a fast way to compute histograms for high-dimensional large datasets?

Currently the way I compute histograms for data is by generating grid in $N$ dimensions (where $N$ is the dimension of the data) and searching through the $M$ data points in each dimension to see in ...
4
votes
2answers
240 views

Hardware random number generator Vs. Pseudo random number generator in the battlefield of Markov Chain Monte Carlo processes

I'm implementing a Markov Chain Monte Carlo process for a Quantum Monte Carlo routine, in every book and paper I've read so far the success of the routine and quality of the results strongly depends ...
4
votes
1answer
2k views

TypeError from scipy.optimize.curve_fit

I am trying to fit a data set to an exponential model using scipy. However, the covariance matrix that is returned is 'inf' and I receive the following error: Traceback (most recent call last): ...
3
votes
1answer
62 views

How to obtain the minimum set of variables required in a model to produce accurate estimation?

I have a system which I assume is linear. I have a matrix $A$ of which each row is a coefficient of a unknown variables in vector $x$. I have vector $B$ which contains the result of each $Ax$. ...
3
votes
0answers
34 views

An asynchronous version of the Covariance Matrix Adaption (CMA)?

Does anyone know of a variant of the CMA algorithm that is suitable for an asynchronous parallel implementation? The conventional version of the algorithm allows one to evaluate the fitness function ...
11
votes
1answer
252 views

Statistical models for local memory/compute, network latency, and bandwidth jitter in HPC

Parallel computation is frequently modeled using a deterministic local rate of computation, latency overhead, and network bandwidth. In reality, these are spatially variable and non-deterministic. ...
1
vote
1answer
324 views

Clustering data based on the radius of a sphere

I have a data-set that has four columns [X Y Z C]. I would like to find all the C values that are in a given sphere centered at [X, Y, Z] with a radius r. What is the best approach to address this ...
2
votes
1answer
218 views

$k$-Nearest Neighbor Search using examples

I want to perform $k$-Nearest Neighbor Search in multidimensional space, but not using for example $L_2$-distance. I want the user to specify some "similar"-pairs examples and then perform a search ...
1
vote
1answer
138 views

Calculating AIC (in R or any other statistical software)

I am trying to fit the log-log plot of the cumulative distribution of a network to one of three models: Exponential (EXP) ($P(k)\sim e^{-ak}$), Exponentially truncated power law (TRU) ($P(k)\sim k^{a-...
3
votes
1answer
160 views

How to add perturbation to a base state in physical grid space?

For study 3D instabilities problem i.e a base state (velocity field and temperature field) and an additional perturbation. The method must be independent of the type of instabilities considered but ...
8
votes
3answers
240 views

How to estimate the impact of small scale on large scale in fluid dynamics?

Assuming that a direct numerical simulation is performed, what is a good method for estimate the impact of small scale on large scale in fluid dynamics ? For example is it pertinent to compare two run ...
2
votes
1answer
200 views

Root Convergence rate of Iterative Scheme

I have an iterative sequence for optimizing an EM (Expectation Maximization) algorithm based loss function $L(X)$ with $t$ being the iteration number as: $X_t=ABX_{t-1}+CX_{t-1}+X_{t-1}$ where $A$ is ...
3
votes
1answer
743 views

In molecular dynamics (MD) simulations, how is particle number density computed in practice?

I have been reading a recent paper. In it, the authors performed molecular dynamics (MD) simulations of parallel-plate supercapacitors, in which liquid resides between the parallel-plate electrodes. ...
5
votes
3answers
913 views

Computing the PDF of a quadratic function of two random variables

Given the function $\mathcal{M} = g + Ah + Bh^2$ where $A$ and $B$ are constants and $g$ and $h$ are random variables with their distributions $f_G(g)$ and $f_H(h)$ known, is it possible to compute ...
0
votes
0answers
167 views

Permissive Math Library for Parameter Statistics in C++

I am looking for recommendations for a C++ math library, with a permissive licence, well suited to calculating a wide variety of statistics on segmentations of timebased parameter data. I would be ...
8
votes
0answers
159 views

Fast algorithms to solve Markov Decision Processes

In my master thesis I used an Algorithm called Approximative Dynamic Programming [1] to solve equations of the form $$ \max_{\pi}\mathbb{E}^{\pi}\left\{\sum_{t=0}^{T}\gamma^tC_t^{\pi}(S_t,A_t^{\pi}(...
29
votes
2answers
26k views

Define custom probability density function in Python

Is there a way, using some established Python package (e.g. SciPy) to define my own probability density function (without any prior data, just $f(x) = a x + b$), so I can then make calculations with ...
5
votes
4answers
561 views

How can I reduce the error of the sample average?

I want to use the sample average $(X_1 + .... X_n)/n$ as a substitute for the expectation $\mathbb{E}(X)$. As claimed by the weak law of large numbers, as n increases the sample average should ...
3
votes
1answer
792 views

Computing a rolling quantile

An algorithm I'm writing needs to compute rolling quantiles of a time series. Currently I do this in the naive way: for a window of size W and a vector ...
3
votes
1answer
66 views

Adding deliberate imperfection to RNG output - toolkits?

Are there any existing software toolkits, libraries, frameworks or whatever for studying the quality of pseudorandom number generators that allow one to add a known amount of imperfection - e.g ...
25
votes
4answers
8k views

How to add large exponential terms reliably without overflow errors?

A very common problem in Markov Chain Monte Carlo involves computing probabilities that are sum of large exponential terms, $ e^{a_1} + e^{a_2} + ... $ where the components of $a$ can range from ...
5
votes
0answers
135 views

Probabilistic algorithms for matrix approximation

Considering regular matrix approximation inequality || $A - QQ^TA $|| < e where we try to approximate matrix $A$ by a lower rank orthonormal matrix $Q$. I've read an article on probabilistic ...
7
votes
4answers
268 views

Testing for stability of a simulated dynamical system

Background and question I often work with simulations of dynamical systems and I usually track a single parameter $x$, such as the number of agents (for agents based models) or the error rate (for ...
8
votes
2answers
1k views

Kolmogorov–Smirnov test for multivariate data

I have a set of files consisting of randomly selected points from a dataset, each file belonging to a particular class. Each row in these files contains the coordinates in n-space of the point. I'd ...
7
votes
2answers
7k views

Looking for C/C++ implementations of sampling from multinomial and Dirichlet distributions

I'm looking for C/C++ implementations of functions that return random variates multinomial and Dirichlet distributions. This is in the context of a calculation for posterior predictive p-values, part ...
4
votes
1answer
918 views

Sampling from posterior predictive distribution

First post. I'm working on this problem using Bayesian methods. In desperation I'm considering using p-values (shock horror), specifically posterior predictive p-values. So I need to simulate from the ...
4
votes
4answers
358 views

Approximate a distribution function from a finite sample

Say I have a simulation that produces a single floating point number as a result, and a different number is produced each time the simulation runs. These numbers are randomly distributed according to ...

1
2