The study of collection, organization, analysis, and interpretation of data.
0
votes
0answers
26 views
Comparing Estimates of Entropy with Theoretical of Multivariate Gaussian
I am trying to compare the effectiveness of a histogram for density estimation, by seeing how well it approximates the PDF of a normal distribution.
I make use of the concept of entropy, which is ...
2
votes
0answers
24 views
Identifying the time window at which a meteorological factor influences tree growth the most
This is a problem I am encountering in a research project. I have thought of a naive approach to solve it which relies on many assumptions, is not guaranteed to work, and would be tedious and ...
0
votes
0answers
10 views
Exotic stats problem mode fuzzy clusters python
EXOTIC STATS PROBLEM ; MODE, FUZZY CLUSTERS, ETC
Hi I'm working on a math problem and not sure how to approach it.
Trying to enumerate, rank and extract most common numeric ranges
from a list, with ...
0
votes
0answers
37 views
How many repititions of a stochastic algorithm should be done before the results can be considered trustworthy?
I have written a stochastic machine learning algorithm.
Each time I run the algorithm on the same test data I get different results. I determine accuracy each time compared to ground truth.
In ...
0
votes
0answers
56 views
Finding correlations between variables in large SQLite database
I have an SQLite3 database file of ~150GB on an external hard-drive, and I would like to analyse correlations (if they exist) between columns of that database. To be precise, I'm interested in ...
0
votes
1answer
54 views
Is there any robust criteria for this kind of outlier?
I have this data, and I need to detect the outliers.
The outliers are clearly visible on the borders (start and end, in red). And I only care for outliers present at the start or end.
I could easily ...
1
vote
0answers
24 views
What is a performant clustering algorithm for approx 10,000 vectors of approx 30 dimension?
I have a set of real-valed vectors, for example
$S = \{v_1, v_2, ..., v_k\}$
$v_i = \begin{pmatrix} age_i \\ height_i \\ weight_i \\ ... \end{pmatrix}$
or whatever. Each vector has on the order of ...
0
votes
0answers
19 views
Uniform distribution in 3D space [duplicate]
Posted this at math stack exchange as well, but alas no replies! So, I have been trying to find ways of distributing particles of spherical or other shape in 3D space, e.g. rectangular space. Random() ...
5
votes
1answer
129 views
Using the PAST algorithm to find eigenvectors
I'm working on trying to extract the eigenvectors from a series of observations from a random variable, by using the PAST algorithm, see e.g. 6.2.3 in this book:
Large pdf.
I don't understand the ...
3
votes
1answer
39 views
Verifying that ODE integration generates Theoretical Stationary distribution
I am trying to simulate an ODE, like $ \dot{x} = \xi(x) $ that should have a stationary distribution (a la Stat Mech). Assuming that my ODE algorithm generates time samples of my system state $ x $ ...
0
votes
2answers
58 views
will this methodology end up giving me a nonsense regression equation.
I'm wondering if this is a valid methodology to find the best regression equation for a given data set.
User provides a rang of estimated value for some set of variables. Th algorithm uses the ...
0
votes
2answers
167 views
Data for Tracy-Widom distribution
I have data for a random variable and I wish to test whether it conforms to the Tracy-Widom distribution. However, the T-W distribution is hard to compute. Is there a readly available table of its ...
0
votes
0answers
66 views
Inbuilt function in C++ to return the value of $c$ in from the CDF $F(c)$ in the case of $t-$ distribution?
Actually I was coding for calculating the confidence interval for mean $\mu$ of the Normal Distribution with unknown $\sigma^{2}$.
In between the algorithm I have to calculate $c$ in $F(c) = \frac{1 +...
1
vote
0answers
123 views
1d turbulent energy spectra in homogenuous direction (non-isotropic)
I am trying to compute the one-dimensional energy spectra for my channel-flow simulation. I have already written a post-processing script to achieve this, however I need to validate my code before ...
3
votes
1answer
65 views
Legendre expansion of $r(x) = f(x)/g(x)$ using a finite number of samples from $f(x)$ and $g(x)$
I have two finite sets of events $\{x_1, ..., x_N\}$ and $\{y_1, ..., y_N\}$ that are sampled from the PDFs $f(x)$ and $g(x)$, respectively, where $x \in [-1,+1]$. I want to estimate the Legendre ...
1
vote
1answer
224 views
Data Analysis - Cooling Efficiency
I have a question as I am starting my dive into computational analysis.
I have a large set of data (~2 months) which includes the room temperature, HVAC status (heat/cool/off) and the location of ...
1
vote
1answer
28 views
Parameter identification for regression model
Consider the following regression model :
Y = AX + BU
where the size of Y is $N \times n$, A is $N \times n$, X is $n \times n$, B is $N \times n$ and U is $n \times 1$.
The matrices X,Y and U are ...
2
votes
1answer
38 views
Using physical parameter as a Gaussian Random Variable in a simple Poisson Problem
I want to vary the input parameter of a physical dynamic mechanics problem, as a Gaussian Random variable and view the resulting Probability Density Function. I used the Finite Element Method to ...
2
votes
1answer
135 views
Optimize custom probability distribution in Python [closed]
Consider random variables $X$ and $Y$, their distributions are given. $Z = f_a(X, Y)$ where $f(\cdot, \cdot)$ is a deterministic, not random function $f_a: \mathbb{R}^2 \to \mathbb{R}$ depending on a ...
7
votes
1answer
136 views
Tikhonov (Ridge) Regression and Normalization
For a typical Ridge Regression method for solving an inverse problem
$$
\min_x ||A~x - b||^2 + \lambda^2||\Gamma~x||^2
$$
Which has an analytical solution of
$$
\hat{x}_{est}=(A^TA+\lambda^2 \Gamma^T\...
1
vote
2answers
38 views
How are the outcomes that generated from different predictive models combined to get more accurate predictions?
The simple average is commonly used to combine the predictions out of different predictive models. Apart form the simple average, what are the other methods that can be used for combining the ...
6
votes
0answers
151 views
Is resampling more accurate than block average for statistical analysis of data?
I'm working in laboratories where molecular dynamics data are almost always analysed usign block average as stated in the famous Allen and Tildesley book.
We divide the datas in blocks of size $M$ on ...
1
vote
0answers
350 views
Generating Random Numbers in Fortran for Metropolis method
I am having a really hard time getting any kind of reliable / consistent result from my Metropolis code. I have torn it apart and am now examining just the randomness in my random number generator.
I ...
0
votes
1answer
37 views
Force a line through the origin
What does it mean: to force a line through the origin?
I interpret this to be making (forcing) the intercept as (0,0) in the regression procedure of an x-y scatterplot, but am not sure.
0
votes
1answer
1k views
Normalize data so that the sum of squares = 1
In presenting geochemical data, I would like to try a statistical method that presents the data in an ISOCON diagram. This method requires scaling all the data to be the same distance from the origin (...
1
vote
0answers
90 views
How to get mathematical model from a data set with MATLAB
I have some values imported from excel about the annual sell of a product, with 3 var:
month,
price,
sold items
I have seen how to plot a chart using curve fitting tool but I need to define a ...
6
votes
2answers
194 views
Sum over very small exponentials: Underflow
I am trying to compute (in C) a sum like
$S = \sum_i \exp( - a_i )$,
where $10^{4} < a_i < 10^{5}$ are approximately normal distributed. So even if I do the Log-Sum-Exp trick
$S = \exp(\...
0
votes
1answer
40 views
Conditional Independence (Bayesian Network)
suppose you have this bayesian network: p(a,b,c) = p(c|b) p(b|a) p(a)
a -> b -> c
are a and c conditionally independent given b?
If yes, why are they independent? How can I show that?
2
votes
1answer
402 views
Is there a relationship between the covariance matrix and the partial derivative?
Suppose that there are $N$ pieces of data, each of which contains $M$ parameter values such that $N >> M$. If we put this information into matrix form ($N$ rows, $M$ columns) and then compute ...
1
vote
2answers
295 views
Polynomial approximation
Is there any universal method to fill this matrix for any $n$ value:
$\textbf{A} = \left[ \matrix{n & \sum x_i & \sum x_i^2 & \cdots & \sum x_i^n \cr
\sum x_i & \sum ...
2
votes
1answer
103 views
CFD turbulence modelling mean pressures vs peak pressures
I have been experimenting with using Autodesk CFD to investigate facade/ cladding pressures on a (rectangular) building, comparing results with cladding/ facades pressures pressures from design codes ...
-1
votes
1answer
124 views
Problem using MATLAB `fminunc` [closed]
I am trying to find the minimum of this function. But I receive the following error when I run the script. What am I doing wrong:
...
1
vote
0answers
42 views
Efficient calculation for L-Kurtosis?
I am doing some statistical signal analysis and was wondering if there are any C/Java packages that do L-moment calculations, specifically L-Kurtosis as I am wanting to do things such as ...
0
votes
1answer
77 views
Can Box-Cox transformation be applied for data of this form?
I have data of the form:
X Y
3.53 0
4.93 50
5.53 60
6.21 70
7.37 80
9.98 90
16.56 100
And I want to find out $n$ so that this can be ...
1
vote
0answers
87 views
How to curve fit an unknown function?
I have data which can be described by $y=f(x,z)$ where $z$ varies from 170 ~ 154. Now values given by $ks$ are known sample values that equals value given in the table header, $uks$ are unknown ...
0
votes
1answer
56 views
Best path for estimation
I have a Cartesian grid (100x100) in which some of the points are known (30 out of 10,000) and the rest are unknown. I want to use the known points and estimate the other cells. Is there any ...
2
votes
1answer
431 views
How to prevent overflow and underflow in the Euclidean distance and Mahalanobis distance
I was working in my project when I was struck by the question of whether it would be necessary, or at least cautious, prevent overflow and underflow in the calculation of these two distances.
I ...
1
vote
0answers
139 views
How to choose (and many) threshold in Eigenface?
I edited my question trying to make it as short and precise.
I am developing a prototype of a facial recognition system for my Graduation Project. I use Eigenface and my main source is the document ...
0
votes
2answers
159 views
Statistical analysis of optimization algorithms
If we optimize some parameter using 4 optimization algorithms, 2 of which are population based (say A and B) and 2 trajectory methods (single point search)(say C and D); what statistical test can be ...
2
votes
2answers
112 views
fastest and most efficent way to count all combinations in many sets and sum them together
I am a Java programmer who has reached the limits of brute computer power. My relational database (and non relational databases) is not producing results quick enough and I have hit a bottleneck in ...
1
vote
1answer
210 views
What is the algorithm that matlab used in its built-in function 'pca'?
Do anyone know what is the algorithm that MATLAB used in its built-in function "pca"?
I have the following data set:
148.9820 55.8438 210.2150
149.3030 56.8891 208.4280
151.4400 ...
0
votes
2answers
688 views
What is 'SOLVER' in R and Statistics/Analytics ?
**strong text**a) I tried to research on what exactly is a SOLVER only to find a not clear-cut simple answer. My doubts still remain after going through several sites full of discussions about it.
I ...
2
votes
2answers
261 views
Python: What is a good way to generate a 1D particle field with a gaussian distribution?
If I have N particles how do I assign their x values so that the end result is Gaussian distribution. i.e. particles near the ends are more spread out than particles near the center.
1
vote
1answer
605 views
Coding complex equations into C++
I have here equations from a paper by E. Bradlow. They're for counting the events in a Weibull-distributed data set.
$$\begin{align}
\Pr(N(t)=n) &= \sum_{j=n}^\infty{\frac{(-1)^{j+n}(\lambda t^c)^...
12
votes
3answers
149 views
Correct statistics for reporting speedup results
Say I have slow and fast versions of some code, and want to report a speedup number comparing the two. I run the slow version $n$ times and the fast version $m$ times, producing times $(s_1, \ldots, ...
1
vote
3answers
147 views
Quality of randomness in generated random number
I have generated a list of 3 random number where each summed to 1. I would like to assess the quality of randomness. What is the best mechanism to assess this randomness? E.g my random numbers are. ...
4
votes
1answer
86 views
Is it possible to compute quantiles of a set of numbers, without first sorting those numbers?
I apologize in advances if this is the wrong place to be asking this question, (I had considered putting it on CV first).
I am studying financial (S&P, Dow Jones, etc) data, and would like to ...
3
votes
1answer
303 views
Difference powerlaw, lognormal and streteched exponential (Weibull) function
I am currently fitting above mentioned functions to my data and I can observe, that both lognormal and Weibull are better fits than powerlaw. In literature, it is often suggested, that it is hard to ...
7
votes
1answer
177 views
Generation of variable with given auto-correlation function
How can I generate realizations of random complex variable $x(t)$ with a given autocorrelation function $C(s)$, defined by
$$C(s) = \langle x(s) x(0) \rangle$$
and obeying the condition $C(-s) = C^*(...
1
vote
1answer
2k views
Information from residuals in polynomial regression
I have two physical properties which are exact (no noise). If I perform a polynomial regression of any degree, all coefficients are statistically significant. The problem is that the residuals are ...
