Questions tagged [data-analysis]
Data Analysis is the process of evaluating data using analytical and logical reasoning to examine each component of the data provided.
72
questions
1
vote
1
answer
157
views
Storing Raw Simulation Data or Truncated Data?
I have a simulation that can generate quite a bit of data when it runs, for example $650\cdot 400 \cdot 400$ floating point numbers. Without compression, that's a few gigabytes worth if I want to save ...
- 33
0
votes
1
answer
36
views
Dimensionality reduction between discrete wavelet families
I have what it may be a ridiculous question (since I don't know much about wavelets), but here I go.
I am using different Discrete Wavelet families to extract texture features from images. I plan to ...
- 3
1
vote
1
answer
84
views
What are the Exact Rules for Significant Figures, Precision, and Uncertainty?
In the physical sciences (which are physics, chemistry, astronomy, materials science, etc.), we learned that the uncertainty is +/- the smallest unit (which is 1) of the last significant figure if the ...
-1
votes
1
answer
123
views
Fitting gauss-hermite-parametrization to data?
I want to fit this data.
I have the following model functions. Classic gaussian:
def gauss_model(x, mu, sigma):
return np.exp(-0.5*((x-mu)/sigma)**2)
And ...
- 99
3
votes
1
answer
46
views
Bounds condition for IFT to obtain a $1/f$ time-series
I am coding a function to obtain a randomized time-series from a given $\frac{1}{f}$ law.
The randomization is obtained by introducing a random phase in the function.
I experience a problem in the ...
- 61
2
votes
1
answer
281
views
-1
votes
1
answer
34
views
Parameter explained by many distributions
If we had, for example, labeled data, where for each entry (label) we have several data distributions associated to it, how can I get something meaningful from them?
Is this a solvable problem? Is ...
- 1
0
votes
0
answers
23
views
I need help developing a process to map many individual records (from a database) into larger groups
There are many rows of data in a database table, each of which contains many columns.
ID (numeric - 12345, 12378, etc)
Name 1 - AAA,BBB,CCC
Name 2 - dd, ee, ff
Name 3 - Ggg, Hhh, Iii
The objective ...
-1
votes
1
answer
46
views
Trouble Estimating Motor Parameters with Least Squares in MATLAB
Basically, I'm trying to use Least-Squares to estimate the parameters of a DC motor.
My system can be modeled by the following matrix equation:
$$\begin{bmatrix}V_{input}(t)\\0\end{bmatrix}=\begin{...
- 109
2
votes
2
answers
240
views
How to extract connected components from persistence diagram?
From the given point cloud (Fig. 1), I use Scipy-TDA to extract persistence diagram (Fig. 2). What I'm interested in is to extract 3 circles. For example, I'd like to know 3 center points and labels ...
- 225
1
vote
1
answer
139
views
Filtering out outliers in a vector field [closed]
I have a vector field that represents a incompressible fluid flow (ie. divergence-free, ideally) that contains a certain percentage of vectors that are completely incorrect, due to the procedure used ...
- 19
1
vote
1
answer
376
views
Spline regularization
I am fitting some B-splines to data, but the data has a "gap" region where the spline is less constrained by the data. I want to devise a regularization scheme to help prevent the spline from ...
- 1,038
1
vote
0
answers
91
views
Eigenfaces Algorithm
This might be a silly quesntion but recently I've been trying to program the eigenface algorithm using PCA, so I arranged the face vectors vertically in a matrix X such as: X = [x1,x2,x3,...,xn];
In ...
- 31
1
vote
1
answer
99
views
Best way of storing numerical data in a compact manner, while leaving it accessible for tools like GnuPlot?
My simulation, written in C++, generates a large amount (roughly ~500) of text files for each set of parameters I try to simulate, with four columns of ~5k double values in each file. Furthermore, to ...
- 543
2
votes
1
answer
257
views
How to check experimental data against a theoretical curve? (Python)
I am trying to check the agreement of a dataset against a theoretical curve, specifically a bandstop filter in an RLC circuit.
I have generated a function which describes the curve we expect from the ...
- 21
2
votes
0
answers
2k
views
Computing the structure factor from positions and radial distribution function [closed]
I'm currently analysing some spatial point patterns that come from some fluid dynamics simulations and I'm having some difficulty computing the structure factor, $S(\pmb{k})$, from both the positions ...
- 183
0
votes
1
answer
127
views
How to improve the efficiency of periodicity detection for long time based lined and gapped datasets
Our data set has $10^4$ data points, but has a long baseline and many gaps. As the histogram shows, the horizontal-axis is time and most of the time, there are no data. The vertical-axis is data ...
- 101
2
votes
3
answers
370
views
Clustering with points lying along different 3D planes
I have a bunch of data points in 3D that lie along a few planes. What would be the best approaches to estimate the normals of these planes?
Edit: There are roughly equal number of points lying along ...
- 29
1
vote
1
answer
67
views
Is there a better way to do run time analysis than this?
I currently have 2 different functions with options to vectorise them: acc_rej_sine(max_iter, algorithm=None) and ...
- 11
3
votes
1
answer
123
views
GPGPU/FPGA programming for Combinatorial Analysis
Recently, I have taken an interest in performing combinatorial analysis for the game of 21 (blackjack) and attempted to use my AMD APU to try and thread the program via the 4 cores on the chip. ...
- 31
0
votes
1
answer
594
views
Converting ROOT Tree to HDF5
I have a TTree in ROOT with 1000 events and 15 variables associated to each of them. I would like to convert this in its entirety to an hdf5 dataset. How do I organise my data in HDF5 Groups such that ...
- 1
2
votes
1
answer
238
views
Which statistical method should I use for comparing machine run-time of two algorithms?
I am comparing the run-time of two algorithm by solving different instance of the problems. Sample of my data:
...
- 23
2
votes
0
answers
31
views
Identifying the time window at which a meteorological factor influences tree growth the most
This is a problem I am encountering in a research project. I have thought of a naive approach to solve it which relies on many assumptions, is not guaranteed to work, and would be tedious and ...
- 121
1
vote
0
answers
33
views
Validating a Markov chain and Bagging to find overall model
let´s say I estimated a Markov chain by splitting my data into traning and test data, and estimating the transition matrix for the traning data yields
$$ P=
\begin{bmatrix}
p11 & p12 &...
0
votes
1
answer
101
views
How much in depth should I cover Python? [closed]
I'm an undergrad who's studying electronics and fields like data sciences and mostly computing seem most interesting to me but have in-depth knowledge in neither. I learnt basic python last winter ...
- 109
1
vote
0
answers
129
views
Implementing Frechet Distance on pandas data frame [duplicate]
I am working on a trajectory analysis project using python and its data science related libraries.
I am planning to implement Frechet Distance algorithm for trajectory analysis, each trajectory has ...
- 121
1
vote
2
answers
4k
views
Python implementation for Frechet Distance
I am working on a trajectory analysis project using python and its data science related libraries. I am planning to implement Frechet Distance algorithm for trajectory analysis, each trajectory has ...
- 121
1
vote
0
answers
31
views
Use custom defined metric(not matrix) in Javaplex
I'm about to get started trying to use Javaplex for the first time for some application in topological data analysis (TDA) and I want to use a custom defined metric. From looking over the documents, I ...
- 111
9
votes
1
answer
7k
views
What is the difference between Computational Science and Data Science? [closed]
Background: My PhD was in 'Computational Science'. My dissertation was on the analysis of X-Ray Diffraction Data and analysis of thermally perturbed nuclei in the overall dynamic analysis of the ...
- 2,119
0
votes
2
answers
43
views
How are the outcomes that generated from different predictive models combined to get more accurate predictions?
The simple average is commonly used to combine the predictions out of different predictive models. Apart form the simple average, what are the other methods that can be used for combining the ...
- 111
1
vote
1
answer
618
views
FFT (Numerical Recipes in FORTRAN 77)
I am trying to use the subroutine twofft which in turn uses subroutine four1 in the book(Numerical Recipes in Fortran) to ...
- 93
6
votes
1
answer
496
views
Is resampling more accurate than block average for statistical analysis of data?
I'm working in laboratories where molecular dynamics data are almost always analysed usign block average as stated in the famous Allen and Tildesley book.
We divide the datas in blocks of size $M$ on ...
- 323
3
votes
1
answer
1k
views
Similarity of two CSV files (or more?)
A CSV file is characterized by a header, describing the n datarows to come. The header is a text string separated by a separator. A CSV header might look like (C1)
...
- 201
1
vote
1
answer
71
views
library for arithmetic operations on unstructured xyz
I've got several sets of xyz-data where x,y are coordinates and z is a 'value'. I want to be able to do simple arithmetic operations on these data. The x,y locations are not necessarily the same in ...
- 113
3
votes
1
answer
424
views
what is the best theory/model to use for prediction in multivariate data?
I use software for pollutant propagation on rivers that takes as input a set of parameters ($p_1,p_2,\ldots,p_n$) and creates an output file which is basically a matrix where on each row the ...
1
vote
1
answer
83
views
Finite difference for Lighthill source term
I have experimental data containing horizontal and vertical components of speed and I need to evaluate this:
$$
\frac{\partial^2}{\partial x_i \partial x_j}\left(v_iv_j\right)
$$
Denoting $U$ as ...
- 145
4
votes
1
answer
3k
views
Is there a relationship between the covariance matrix and the partial derivative?
Suppose that there are $N$ pieces of data, each of which contains $M$ parameter values such that $N >> M$. If we put this information into matrix form ($N$ rows, $M$ columns) and then compute ...
- 2,119
3
votes
1
answer
262
views
similarity/distance measurement between two ranked sequence
Is there an efficient way to measure similarity/distance between two sequences of ranked numbers/letters. The two sequences are of different length, and only have some elements in common?
For example,...
- 63
1
vote
0
answers
45
views
Manipulating/Extracting Data and Developing Methods - Language Choice [closed]
As a general programming enthusiast and aspiring Bioinformatician student I have an intermediate understanding of computing (languages) as well as Java, and to a lesser extent C++. Having knowledge in ...
- 11
5
votes
1
answer
342
views
Iterative Cunnigham correction in Millikans oil-drop experiment
I tried to write an analysis program in python for the Millikan experiment based on a paper from Jones "The Millikan oil‐drop experiment: Making it worthwhile"
However, I cannot reproduce the ...
- 55
0
votes
1
answer
666
views
Power series regression linear fit in VBA excel
I wrote a program that calculates the best fit in VBA excel for the following model
$$
y_k=c_1x_k+c_0+c_{-1}(x_k)^{-1}
$$
solving for the best fit parameters $c_1$, $c_0$, and $c_{-1}$. However I ...
- 113
2
votes
1
answer
47
views
Simple Path to Route Algorithm
I'm currently conducting research into subway paths, but stand in front of a problem, with no programming knowledge and an extremely short time-limit.
Probably the question I am asking is trivial, ...
- 21
0
votes
1
answer
159
views
Can Box-Cox transformation be applied for data of this form?
I have data of the form:
X Y
3.53 0
4.93 50
5.53 60
6.21 70
7.37 80
9.98 90
16.56 100
And I want to find out $n$ so that this can be ...
- 119
0
votes
1
answer
3k
views
Gonzalez algorithm
I'm getting confused about implementing the Gonzalez algorithm for k-center clustering.
The algorithm is :
let $S$ be a data set
Choose $z_1 \in S$ arbitrarily. Number $z$ is center. (In general ...
- 21
0
votes
2
answers
939
views
What is "SOLVER" in R and Statistics/Analytics?
I tried to research what exactly is a SOLVER only to find a not clear-cut simple answer. My doubts still remain after going through several sites full of discussions about it.
I need to clarify as to ...
- 3
5
votes
2
answers
338
views
Dimensionality reduction of the domain of f(x)
I'm wondering if there is something analogous to a PCA for data sets where there is a dependent variable. (Though I am interested in any method of dimensionality reduction, PCA is just an example.) ...
- 153
3
votes
1
answer
171
views
Monte Carlo Simulation - Random Number Motivation
For Monte Carlo simulations, or any other numerical methods that rely heavily on the quality of the pseudo-random numbers generated (i.e even/desired distribution on a certain domain) for that matter, ...
- 489
8
votes
4
answers
721
views
Should I use a database to handle large amounts of results?
Background:
I am currently running a large amount parameter variation experiments.
They are being run in Python 2.6+, using numpy.
These experiments are going to take about 2 weeks to run.
Roughly I ...
1
vote
0
answers
152
views
SciDB vs OLAP cube [closed]
I am working on good retrieval mechanism for big data systems. I am of view that multidimensional structures could be of great use for aggregations and querying . Multidimensional structures as ...
- 11
2
votes
2
answers
373
views
Extracting time scales information from empirical cumulative distribution function
I have a stochastic process (a Markov chain actually) that has two absorbing states. I am using a difference equation to calculate the first passage time to either of the absorbing states. There are ...
- 279