Questions tagged [statistics]

The study of collection, organization, analysis, and interpretation of data.

Filter by
Sorted by
Tagged with
2
votes
1answer
558 views

1-D turbulent energy spectra in homogeneous direction (non-isotropic)

I am trying to compute the one-dimensional energy spectra for my channel-flow simulation. I have already written a post-processing script to achieve this; however, I need to validate my code before ...
-1
votes
1answer
36 views

Generating Rvs for a given PDf in python

Two random variables $X$ and $Y$ are distributed according to \begin{align} f_{XY}(x,y)= \begin{cases} x+y & 0\leq x \leq 1, 0\leq y \leq 1 \\ 0 & otherwise \end{cases} \end{align} I ...
4
votes
0answers
58 views

Efficient computation of marginalized multivariate normal likelihood

In general,if we know that the marginal Gaussian distribution for some variable $\textbf{x}$ and a conditional Gaussian distribution for some $\textbf{y}|\textbf{x}$ of the forms: $$p(\textbf{x}) = \...
1
vote
1answer
72 views

Define continuous, non-analytical pdfs in python

I am planning to do some basic algebra on continuous, non-analytical random variabels. I want to define their probability density functions as arrays x and f(x). Yet, I was surprised to find out that ...
3
votes
2answers
204 views

How to optimize sampling for parameter estimation

I have a computer model with a number of parameters that need to be calibrated based on experimental results. It's also important to understand the sensitivity of the results to each parameter ...
4
votes
1answer
147 views

How to define a dimensionless Objective function for determining how peaked a curve is?

I have attached 2 plots for FFT spectra. One is considered good and one is bad. The good one is classified on the basis of how closely spaced the frequencies and the bad is based on how multiple ...
2
votes
0answers
66 views

Whi are chi-squared distributions in (C++) boost::random and C++ STL faster than in boost::math?

I am trying to generate random chi-squared numbers in C++, according to some degree of freedom (which can be a float). Several libraries can be used for that purpose, among which the C++11 Standard ...
2
votes
1answer
95 views

Best way to convert a sparse (containing zeros) covariance matrix into a correlation matrix?

I have a $100$x$100$ covariance matrix that looks like this. Some rows/cols are all-zero because those corresponding elements are not present in the sample from which covariance is calculated. I'm ...
2
votes
0answers
51 views

Partial/Extended/Truncated Template Matching

So template matching using correlation is available in a lot of computational packages; OpenCV matchTemplate(), scipy.signal.correlate2d(), IPP CrossCorrNorm, etc. But they all either evaluate ...
3
votes
1answer
56 views

Reconstructing statistics of $x\otimes y$ from E[XX'], E[YY'] and E[XY']

I'm looking at random vectors $z$ of size $d^2$ which can be written as $z=x\otimes y$ where $x$,$y$ are random vectors in $\mathbb{R}^d$ with following second moments known -- $E[XX']$, $E[YY']$ and $...
6
votes
1answer
170 views

Computing square root of diag(u)-uu'?

I need an efficient way to take square root of a matrix which is a sum of diagonal matrix and rank-1 matrix. More specifically it's the following matrix $$A=D-uu'=\text{diag}(u)-uu'$$ Where entries ...
2
votes
1answer
152 views

Which statistical method should I use for comparing machine run-time of two algorithms?

I am comparing the run-time of two algorithm by solving different instance of the problems. Sample of my data: ...
1
vote
1answer
381 views

Weighted moving variance

i have a time-series and, in analogy with exponentially weighted moving average, i would like to compute the exponentially weighted moving standard deviation or variance in an efficient, numerically ...
1
vote
0answers
42 views

Fast convergence of smoothing of periodic noise

I have essentially periodic data from a simulation (not exactly periodic but is qualitatively fairly periodic), and I'd like to take an average or noise filter of some sort that I can get a well ...
1
vote
0answers
92 views

difference between PLSA and LDA

What is the difference between Probabilistic Latent Semantic Analysis(pLSA) and Latent Dirichlet Allocation (LDA)?
2
votes
1answer
62 views

Using physical parameter as a Gaussian random variable in a simple Poisson problem

I want to vary the input parameter of a physical dynamic mechanics problem, as a Gaussian Random variable and view the resulting Probability Density Function (PDF). I used the Finite Element Method to ...
2
votes
1answer
64 views

Efficient calculation for L-Kurtosis?

I am doing some statistical signal analysis and was wondering if there are any C/Java packages that do L-moment calculations, specifically L-Kurtosis as I am wanting to do things such as ...
0
votes
0answers
30 views

how to estimate the number of people on a street within an hour?

I tried to use opencv to analyze a video filmed on the street. But the problem is the performance is not enough. I think the number of people must follow the poisson distribution. So I want to ...
4
votes
1answer
918 views

Sampling from posterior predictive distribution

First post. I'm working on this problem using Bayesian methods. In desperation I'm considering using p-values (shock horror), specifically posterior predictive p-values. So I need to simulate from the ...
7
votes
4answers
268 views

Testing for stability of a simulated dynamical system

Background and question I often work with simulations of dynamical systems and I usually track a single parameter $x$, such as the number of agents (for agents based models) or the error rate (for ...
1
vote
0answers
71 views

Why does the correlation function of this stochastic differential equation starts at different points?

I am working with the following differential equation: The equation is $$x=\beta +\sqrt{2D} \xi(t)$$ where $\xi(t)$ is a white noise term, with a reflecting wall boundary conditions. After solving ...
6
votes
1answer
361 views

Is resampling more accurate than block average for statistical analysis of data?

I'm working in laboratories where molecular dynamics data are almost always analysed usign block average as stated in the famous Allen and Tildesley book. We divide the datas in blocks of size $M$ on ...
1
vote
1answer
240 views

Data Analysis - Cooling Efficiency

I have a question as I am starting my dive into computational analysis. I have a large set of data (~2 months) which includes the room temperature, HVAC status (heat/cool/off) and the location of ...
2
votes
0answers
31 views

Identifying the time window at which a meteorological factor influences tree growth the most

This is a problem I am encountering in a research project. I have thought of a naive approach to solve it which relies on many assumptions, is not guaranteed to work, and would be tedious and ...
0
votes
1answer
63 views

Is there any robust criteria for this kind of outlier?

I have this data, and I need to detect the outliers. The outliers are clearly visible on the borders (start and end, in red). And I only care for outliers present at the start or end. I could easily ...
4
votes
1answer
157 views

Using the PAST algorithm to find eigenvectors

I'm working on trying to extract the eigenvectors from a series of observations from a random variable, by using the PAST algorithm, see e.g. 6.2.3 in this book: Large pdf. I don't understand the ...
1
vote
0answers
45 views

What is a performant clustering algorithm for approx 10,000 vectors of approx 30 dimension?

I have a set of real-valed vectors, for example $S = \{v_1, v_2, ..., v_k\}$ $v_i = \begin{pmatrix} age_i \\ height_i \\ weight_i \\ ... \end{pmatrix}$ or whatever. Each vector has on the order of ...
1
vote
2answers
327 views

Polynomial approximation

Is there any universal method to fill this matrix for any $n$ value: $\textbf{A} = \left[ \matrix{n & \sum x_i & \sum x_i^2 & \cdots & \sum x_i^n \cr \sum x_i & \sum ...
1
vote
0answers
23 views

Uniform distribution in 3D space [duplicate]

Posted this at math stack exchange as well, but alas no replies! So, I have been trying to find ways of distributing particles of spherical or other shape in 3D space, e.g. rectangular space. Random() ...
2
votes
1answer
44 views

Verifying that ODE integration generates Theoretical Stationary distribution

I am trying to simulate an ODE, like $ \dot{x} = \xi(x) $ that should have a stationary distribution (a la Stat Mech). Assuming that my ODE algorithm generates time samples of my system state $ x $ ...
0
votes
2answers
61 views

will this methodology end up giving me a nonsense regression equation.

I'm wondering if this is a valid methodology to find the best regression equation for a given data set. User provides a rang of estimated value for some set of variables. Th algorithm uses the ...
1
vote
2answers
463 views

Data for Tracy-Widom distribution

I have data for a random variable and I wish to test whether it conforms to the Tracy-Widom distribution. However, the T-W distribution is hard to compute. Is there a readly available table of its ...
29
votes
2answers
26k views

Define custom probability density function in Python

Is there a way, using some established Python package (e.g. SciPy) to define my own probability density function (without any prior data, just $f(x) = a x + b$), so I can then make calculations with ...
2
votes
1answer
70 views

Legendre expansion of $r(x) = f(x)/g(x)$ using a finite number of samples from $f(x)$ and $g(x)$

I have two finite sets of events $\{x_1, ..., x_N\}$ and $\{y_1, ..., y_N\}$ that are sampled from the PDFs $f(x)$ and $g(x)$, respectively, where $x \in [-1,+1]$. I want to estimate the Legendre ...
7
votes
1answer
187 views

Generation of variable with given auto-correlation function

How can I generate realizations of random complex variable $x(t)$ with a given autocorrelation function $C(s)$, defined by $$C(s) = \langle x(s) x(0) \rangle$$ and obeying the condition $C(-s) = C^*(...
1
vote
1answer
34 views

Parameter identification for regression model

Consider the following regression model : Y = AX + BU where the size of Y is $N \times n$, A is $N \times n$, X is $n \times n$, B is $N \times n$ and U is $n \times 1$. The matrices X,Y and U are ...
2
votes
1answer
272 views

Optimize custom probability distribution in Python [closed]

Consider random variables $X$ and $Y$, their distributions are given. $Z = f_a(X, Y)$ where $f(\cdot, \cdot)$ is a deterministic, not random function $f_a: \mathbb{R}^2 \to \mathbb{R}$ depending on a ...
6
votes
1answer
220 views

Tikhonov (Ridge) Regression and Normalization

For a typical Ridge Regression method for solving an inverse problem $$ \min_x ||A~x - b||^2 + \lambda^2||\Gamma~x||^2 $$ Which has an analytical solution of $$ \hat{x}_{est}=(A^TA+\lambda^2 \Gamma^T\...
0
votes
2answers
41 views

How are the outcomes that generated from different predictive models combined to get more accurate predictions?

The simple average is commonly used to combine the predictions out of different predictive models. Apart form the simple average, what are the other methods that can be used for combining the ...
1
vote
0answers
810 views

Generating Random Numbers in Fortran for Metropolis method

I am having a really hard time getting any kind of reliable / consistent result from my Metropolis code. I have torn it apart and am now examining just the randomness in my random number generator. I ...
0
votes
1answer
4k views

Normalize data so that the sum of squares = 1

In presenting geochemical data, I would like to try a statistical method that presents the data in an ISOCON diagram. This method requires scaling all the data to be the same distance from the origin (...
0
votes
1answer
46 views

Force a line through the origin

What does it mean: to force a line through the origin? I interpret this to be making (forcing) the intercept as (0,0) in the regression procedure of an x-y scatterplot, but am not sure.
1
vote
0answers
108 views

How to get mathematical model from a data set with MATLAB

I have some values imported from excel about the annual sell of a product, with 3 var: month, price, sold items I have seen how to plot a chart using curve fitting tool but I need to define a ...
8
votes
2answers
827 views

Sum over very small exponentials: Underflow

I am trying to compute (in C) a sum like $S = \sum_i \exp( - a_i )$, where $10^{4} < a_i < 10^{5}$ are approximately normal distributed. So even if I do the Log-Sum-Exp trick $S = \exp(\...
0
votes
1answer
41 views

Conditional Independence (Bayesian Network)

suppose you have this bayesian network: p(a,b,c) = p(c|b) p(b|a) p(a) a -> b -> c are a and c conditionally independent given b? If yes, why are they independent? How can I show that?
4
votes
1answer
2k views

Is there a relationship between the covariance matrix and the partial derivative?

Suppose that there are $N$ pieces of data, each of which contains $M$ parameter values such that $N >> M$. If we put this information into matrix form ($N$ rows, $M$ columns) and then compute ...
0
votes
1answer
138 views

Can Box-Cox transformation be applied for data of this form?

I have data of the form: X Y 3.53 0 4.93 50 5.53 60 6.21 70 7.37 80 9.98 90 16.56 100 And I want to find out $n$ so that this can be ...
7
votes
2answers
4k views

Is there a fast way to compute histograms for high-dimensional large datasets?

Currently the way I compute histograms for data is by generating grid in $N$ dimensions (where $N$ is the dimension of the data) and searching through the $M$ data points in each dimension to see in ...
2
votes
1answer
120 views

CFD turbulence modelling mean pressures vs peak pressures

I have been experimenting with using Autodesk CFD to investigate facade/ cladding pressures on a (rectangular) building, comparing results with cladding/ facades pressures pressures from design codes ...
-1
votes
1answer
161 views

Problem using MATLAB `fminunc` [closed]

I am trying to find the minimum of this function. But I receive the following error when I run the script. What am I doing wrong: ...