Questions tagged [statistics]

The study of collection, organization, analysis, and interpretation of data.

Filter by
Sorted by
Tagged with
29
votes
2answers
26k views

Define custom probability density function in Python

Is there a way, using some established Python package (e.g. SciPy) to define my own probability density function (without any prior data, just $f(x) = a x + b$), so I can then make calculations with ...
25
votes
4answers
8k views

How to add large exponential terms reliably without overflow errors?

A very common problem in Markov Chain Monte Carlo involves computing probabilities that are sum of large exponential terms, $ e^{a_1} + e^{a_2} + ... $ where the components of $a$ can range from ...
12
votes
3answers
221 views

Correct statistics for reporting speedup results

Say I have slow and fast versions of some code, and want to report a speedup number comparing the two. I run the slow version $n$ times and the fast version $m$ times, producing times $(s_1, \ldots, ...
11
votes
1answer
252 views

Statistical models for local memory/compute, network latency, and bandwidth jitter in HPC

Parallel computation is frequently modeled using a deterministic local rate of computation, latency overhead, and network bandwidth. In reality, these are spatially variable and non-deterministic. ...
8
votes
3answers
240 views

How to estimate the impact of small scale on large scale in fluid dynamics?

Assuming that a direct numerical simulation is performed, what is a good method for estimate the impact of small scale on large scale in fluid dynamics ? For example is it pertinent to compare two run ...
8
votes
2answers
1k views

Kolmogorov–Smirnov test for multivariate data

I have a set of files consisting of randomly selected points from a dataset, each file belonging to a particular class. Each row in these files contains the coordinates in n-space of the point. I'd ...
8
votes
2answers
827 views

Sum over very small exponentials: Underflow

I am trying to compute (in C) a sum like $S = \sum_i \exp( - a_i )$, where $10^{4} < a_i < 10^{5}$ are approximately normal distributed. So even if I do the Log-Sum-Exp trick $S = \exp(\...
8
votes
0answers
159 views

Fast algorithms to solve Markov Decision Processes

In my master thesis I used an Algorithm called Approximative Dynamic Programming [1] to solve equations of the form $$ \max_{\pi}\mathbb{E}^{\pi}\left\{\sum_{t=0}^{T}\gamma^tC_t^{\pi}(S_t,A_t^{\pi}(...
7
votes
2answers
7k views

Looking for C/C++ implementations of sampling from multinomial and Dirichlet distributions

I'm looking for C/C++ implementations of functions that return random variates multinomial and Dirichlet distributions. This is in the context of a calculation for posterior predictive p-values, part ...
7
votes
2answers
4k views

Is there a fast way to compute histograms for high-dimensional large datasets?

Currently the way I compute histograms for data is by generating grid in $N$ dimensions (where $N$ is the dimension of the data) and searching through the $M$ data points in each dimension to see in ...
7
votes
1answer
187 views

Generation of variable with given auto-correlation function

How can I generate realizations of random complex variable $x(t)$ with a given autocorrelation function $C(s)$, defined by $$C(s) = \langle x(s) x(0) \rangle$$ and obeying the condition $C(-s) = C^*(...
7
votes
4answers
268 views

Testing for stability of a simulated dynamical system

Background and question I often work with simulations of dynamical systems and I usually track a single parameter $x$, such as the number of agents (for agents based models) or the error rate (for ...
6
votes
1answer
220 views

Tikhonov (Ridge) Regression and Normalization

For a typical Ridge Regression method for solving an inverse problem $$ \min_x ||A~x - b||^2 + \lambda^2||\Gamma~x||^2 $$ Which has an analytical solution of $$ \hat{x}_{est}=(A^TA+\lambda^2 \Gamma^T\...
6
votes
1answer
170 views

Computing square root of diag(u)-uu'?

I need an efficient way to take square root of a matrix which is a sum of diagonal matrix and rank-1 matrix. More specifically it's the following matrix $$A=D-uu'=\text{diag}(u)-uu'$$ Where entries ...
6
votes
1answer
213 views

What computational methods would allow me to rank 2D surfaces (with examples)

I have a program which compares the similarity of two images for different positions, so my surface consists of points which correspond to X and Y translations each with a value (mutual information). ...
6
votes
1answer
361 views

Is resampling more accurate than block average for statistical analysis of data?

I'm working in laboratories where molecular dynamics data are almost always analysed usign block average as stated in the famous Allen and Tildesley book. We divide the datas in blocks of size $M$ on ...
5
votes
4answers
562 views

How can I reduce the error of the sample average?

I want to use the sample average $(X_1 + .... X_n)/n$ as a substitute for the expectation $\mathbb{E}(X)$. As claimed by the weak law of large numbers, as n increases the sample average should ...
5
votes
3answers
913 views

Computing the PDF of a quadratic function of two random variables

Given the function $\mathcal{M} = g + Ah + Bh^2$ where $A$ and $B$ are constants and $g$ and $h$ are random variables with their distributions $f_G(g)$ and $f_H(h)$ known, is it possible to compute ...
5
votes
0answers
135 views

Probabilistic algorithms for matrix approximation

Considering regular matrix approximation inequality || $A - QQ^TA $|| < e where we try to approximate matrix $A$ by a lower rank orthonormal matrix $Q$. I've read an article on probabilistic ...
4
votes
4answers
358 views

Approximate a distribution function from a finite sample

Say I have a simulation that produces a single floating point number as a result, and a different number is produced each time the simulation runs. These numbers are randomly distributed according to ...
4
votes
2answers
240 views

Hardware random number generator Vs. Pseudo random number generator in the battlefield of Markov Chain Monte Carlo processes

I'm implementing a Markov Chain Monte Carlo process for a Quantum Monte Carlo routine, in every book and paper I've read so far the success of the routine and quality of the results strongly depends ...
4
votes
1answer
147 views

How to define a dimensionless Objective function for determining how peaked a curve is?

I have attached 2 plots for FFT spectra. One is considered good and one is bad. The good one is classified on the basis of how closely spaced the frequencies and the bad is based on how multiple ...
4
votes
1answer
2k views

Is there a relationship between the covariance matrix and the partial derivative?

Suppose that there are $N$ pieces of data, each of which contains $M$ parameter values such that $N >> M$. If we put this information into matrix form ($N$ rows, $M$ columns) and then compute ...
4
votes
1answer
189 views

Is it possible to compute quantiles of a set of numbers, without first sorting those numbers?

I apologize in advances if this is the wrong place to be asking this question, (I had considered putting it on CV first). I am studying financial (S&P, Dow Jones, etc) data, and would like to ...
4
votes
1answer
918 views

Sampling from posterior predictive distribution

First post. I'm working on this problem using Bayesian methods. In desperation I'm considering using p-values (shock horror), specifically posterior predictive p-values. So I need to simulate from the ...
4
votes
1answer
157 views

Using the PAST algorithm to find eigenvectors

I'm working on trying to extract the eigenvectors from a series of observations from a random variable, by using the PAST algorithm, see e.g. 6.2.3 in this book: Large pdf. I don't understand the ...
4
votes
1answer
2k views

TypeError from scipy.optimize.curve_fit

I am trying to fit a data set to an exponential model using scipy. However, the covariance matrix that is returned is 'inf' and I receive the following error: Traceback (most recent call last): ...
4
votes
0answers
58 views

Efficient computation of marginalized multivariate normal likelihood

In general,if we know that the marginal Gaussian distribution for some variable $\textbf{x}$ and a conditional Gaussian distribution for some $\textbf{y}|\textbf{x}$ of the forms: $$p(\textbf{x}) = \...
3
votes
1answer
160 views

How to add perturbation to a base state in physical grid space?

For study 3D instabilities problem i.e a base state (velocity field and temperature field) and an additional perturbation. The method must be independent of the type of instabilities considered but ...
3
votes
1answer
793 views

Computing a rolling quantile

An algorithm I'm writing needs to compute rolling quantiles of a time series. Currently I do this in the naive way: for a window of size W and a vector ...
3
votes
1answer
56 views

Reconstructing statistics of $x\otimes y$ from E[XX'], E[YY'] and E[XY']

I'm looking at random vectors $z$ of size $d^2$ which can be written as $z=x\otimes y$ where $x$,$y$ are random vectors in $\mathbb{R}^d$ with following second moments known -- $E[XX']$, $E[YY']$ and $...
3
votes
1answer
344 views

Difference powerlaw, lognormal and streteched exponential (Weibull) function

I am currently fitting above mentioned functions to my data and I can observe, that both lognormal and Weibull are better fits than powerlaw. In literature, it is often suggested, that it is hard to ...
3
votes
1answer
62 views

How to obtain the minimum set of variables required in a model to produce accurate estimation?

I have a system which I assume is linear. I have a matrix $A$ of which each row is a coefficient of a unknown variables in vector $x$. I have vector $B$ which contains the result of each $Ax$. ...
3
votes
1answer
66 views

Adding deliberate imperfection to RNG output - toolkits?

Are there any existing software toolkits, libraries, frameworks or whatever for studying the quality of pseudorandom number generators that allow one to add a known amount of imperfection - e.g ...
3
votes
1answer
743 views

In molecular dynamics (MD) simulations, how is particle number density computed in practice?

I have been reading a recent paper. In it, the authors performed molecular dynamics (MD) simulations of parallel-plate supercapacitors, in which liquid resides between the parallel-plate electrodes. ...
3
votes
2answers
204 views

How to optimize sampling for parameter estimation

I have a computer model with a number of parameters that need to be calibrated based on experimental results. It's also important to understand the sensitivity of the results to each parameter ...
3
votes
0answers
216 views

How to estimate if a velocity field is statistically homogeneous?

For a 3D velocity field $\mathbf{u}$ obtained by direct numerical simulation. Assuming that the field is defined on a periodic domain $\mathcal{P}$ of periodicity $L_x$ in the $x$ direction, $L_y$ ...
3
votes
0answers
34 views

An asynchronous version of the Covariance Matrix Adaption (CMA)?

Does anyone know of a variant of the CMA algorithm that is suitable for an asynchronous parallel implementation? The conventional version of the algorithm allows one to evaluate the fitness function ...
2
votes
2answers
700 views

Python: What is a good way to generate a 1D particle field with a gaussian distribution?

If I have N particles how do I assign their x values so that the end result is Gaussian distribution. i.e. particles near the ends are more spread out than particles near the center.
2
votes
1answer
44 views

Verifying that ODE integration generates Theoretical Stationary distribution

I am trying to simulate an ODE, like $ \dot{x} = \xi(x) $ that should have a stationary distribution (a la Stat Mech). Assuming that my ODE algorithm generates time samples of my system state $ x $ ...
2
votes
1answer
3k views

Information from residuals in polynomial regression

I have two physical properties which are exact (no noise). If I perform a polynomial regression of any degree, all coefficients are statistically significant. The problem is that the residuals are ...
2
votes
1answer
95 views

Best way to convert a sparse (containing zeros) covariance matrix into a correlation matrix?

I have a $100$x$100$ covariance matrix that looks like this. Some rows/cols are all-zero because those corresponding elements are not present in the sample from which covariance is calculated. I'm ...
2
votes
1answer
152 views

Which statistical method should I use for comparing machine run-time of two algorithms?

I am comparing the run-time of two algorithm by solving different instance of the problems. Sample of my data: ...
2
votes
1answer
200 views

Root Convergence rate of Iterative Scheme

I have an iterative sequence for optimizing an EM (Expectation Maximization) algorithm based loss function $L(X)$ with $t$ being the iteration number as: $X_t=ABX_{t-1}+CX_{t-1}+X_{t-1}$ where $A$ is ...
2
votes
1answer
70 views

Legendre expansion of $r(x) = f(x)/g(x)$ using a finite number of samples from $f(x)$ and $g(x)$

I have two finite sets of events $\{x_1, ..., x_N\}$ and $\{y_1, ..., y_N\}$ that are sampled from the PDFs $f(x)$ and $g(x)$, respectively, where $x \in [-1,+1]$. I want to estimate the Legendre ...
2
votes
1answer
62 views

Using physical parameter as a Gaussian random variable in a simple Poisson problem

I want to vary the input parameter of a physical dynamic mechanics problem, as a Gaussian Random variable and view the resulting Probability Density Function (PDF). I used the Finite Element Method to ...
2
votes
1answer
272 views

Optimize custom probability distribution in Python [closed]

Consider random variables $X$ and $Y$, their distributions are given. $Z = f_a(X, Y)$ where $f(\cdot, \cdot)$ is a deterministic, not random function $f_a: \mathbb{R}^2 \to \mathbb{R}$ depending on a ...
2
votes
1answer
120 views

CFD turbulence modelling mean pressures vs peak pressures

I have been experimenting with using Autodesk CFD to investigate facade/ cladding pressures on a (rectangular) building, comparing results with cladding/ facades pressures pressures from design codes ...
2
votes
1answer
64 views

Efficient calculation for L-Kurtosis?

I am doing some statistical signal analysis and was wondering if there are any C/Java packages that do L-moment calculations, specifically L-Kurtosis as I am wanting to do things such as ...
2
votes
1answer
542 views

How to prevent overflow and underflow in the Euclidean distance and Mahalanobis distance

I was working in my project when I was struck by the question of whether it would be necessary, or at least cautious, prevent overflow and underflow in the calculation of these two distances. I ...