# Questions tagged [data-analysis]

Data Analysis is the process of evaluating data using analytical and logical reasoning to examine each component of the data provided.

64 questions
Filter by
Sorted by
Tagged with
44 views

### Triplet optimization algorithm producing weird results?

I wrote an algorithm for data analysis using the CERN ROOT framework. It takes in three files of sorted UNIX timestamps from a single year and pairs them up in the closest triplets, where each file ...
64 views

### How to extract connected components from persistence diagram?

From the given point cloud (Fig. 1), I use Scipy-TDA to extract persistence diagram (Fig. 2). What I'm interested in is to extract 3 circles. For example, I'd like to know 3 center points and labels ...
72 views

### Filtering out outliers in a vector field [closed]

I have a vector field that represents a incompressible fluid flow (ie. divergence-free, ideally) that contains a certain percentage of vectors that are completely incorrect, due to the procedure used ...
59 views

### Spline regularization

I am fitting some B-splines to data, but the data has a "gap" region where the spline is less constrained by the data. I want to devise a regularization scheme to help prevent the spline from ...
85 views

### Eigenfaces Algorithm

This might be a silly quesntion but recently I've been trying to program the eigenface algorithm using PCA, so I arranged the face vectors vertically in a matrix X such as: X = [x1,x2,x3,...,xn]; In ...
67 views

### Best way of storing numerical data in a compact manner, while leaving it accessible for tools like GnuPlot?

My simulation, written in C++, generates a large amount (roughly ~500) of text files for each set of parameters I try to simulate, with four columns of ~5k double values in each file. Furthermore, to ...
84 views

### How to check experimental data against a theoretical curve? (Python)

I am trying to check the agreement of a dataset against a theoretical curve, specifically a bandstop filter in an RLC circuit. I have generated a function which describes the curve we expect from the ...
515 views

### Computing the structure factor from positions and radial distribution function [closed]

I'm currently analysing some spatial point patterns that come from some fluid dynamics simulations and I'm having some difficulty computing the structure factor, $S(\pmb{k})$, from both the positions ...
63 views

### How to improve the efficiency of periodicity detection for long time based lined and gapped datasets

Our data set has $10^4$ data points, but has a long baseline and many gaps. As the histogram shows, the horizontal-axis is time and most of the time, there are no data. The vertical-axis is data ...
190 views

### Clustering with points lying along different 3D planes

I have a bunch of data points in 3D that lie along a few planes. What would be the best approaches to estimate the normals of these planes? Edit: There are roughly equal number of points lying along ...
62 views

### Is there a better way to do run time analysis than this?

I currently have 2 different functions with options to vectorise them: acc_rej_sine(max_iter, algorithm=None) and ...
70 views

### GPGPU/FPGA programming for Combinatorial Analysis

Recently, I have taken an interest in performing combinatorial analysis for the game of 21 (blackjack) and attempted to use my AMD APU to try and thread the program via the 4 cores on the chip. ...
234 views

### Converting ROOT Tree to HDF5

I have a TTree in ROOT with 1000 events and 15 variables associated to each of them. I would like to convert this in its entirety to an hdf5 dataset. How do I organise my data in HDF5 Groups such that ...
124 views

### Which statistical method should I use for comparing machine run-time of two algorithms?

I am comparing the run-time of two algorithm by solving different instance of the problems. Sample of my data: ...
31 views

### Identifying the time window at which a meteorological factor influences tree growth the most

This is a problem I am encountering in a research project. I have thought of a naive approach to solve it which relies on many assumptions, is not guaranteed to work, and would be tedious and ...
25 views

41 views

### Simple Path to Route Algorithm

I'm currently conducting research into subway paths, but stand in front of a problem, with no programming knowledge and an extremely short time-limit. Probably the question I am asking is trivial, ...
131 views

### Can Box-Cox transformation be applied for data of this form?

I have data of the form: X Y 3.53 0 4.93 50 5.53 60 6.21 70 7.37 80 9.98 90 16.56 100 And I want to find out $n$ so that this can be ...
2k views

### Gonzalez algorithm

I'm getting confused about implementing the Gonzalez algorithm for k-center clustering. The algorithm is : let $S$ be a data set Choose $z_1 \in S$ arbitrarily. Number $z$ is center. (In general ...
831 views

### What is 'SOLVER' in R and Statistics/Analytics ?

**strong text**a) I tried to research on what exactly is a SOLVER only to find a not clear-cut simple answer. My doubts still remain after going through several sites full of discussions about it. I ...
293 views

### Dimensionality reduction of the domain of f(x)

I'm wondering if there is something analogous to a PCA for data sets where there is a dependent variable. (Though I am interested in any method of dimensionality reduction, PCA is just an example.) ...
142 views

### Monte Carlo Simulation - Random Number Motivation

For Monte Carlo simulations, or any other numerical methods that rely heavily on the quality of the pseudo-random numbers generated (i.e even/desired distribution on a certain domain) for that matter, ...
582 views

### Should I use a database to handle large amounts of results?

Background: I am currently running a large amount parameter variation experiments. They are being run in Python 2.6+, using numpy. These experiments are going to take about 2 weeks to run. Roughly I ...
147 views

### SciDB vs OLAP cube [closed]

I am working on good retrieval mechanism for big data systems. I am of view that multidimensional structures could be of great use for aggregations and querying . Multidimensional structures as ...
305 views

### Extracting time scales information from empirical cumulative distribution function

I have a stochastic process (a Markov chain actually) that has two absorbing states. I am using a difference equation to calculate the first passage time to either of the absorbing states. There are ...
68 views

### Finding maximum value in huge dataset

I am working on a project involving some pattern recognition. For this I need to find the maximum value in a huge multidimensional dataset. For example I have a discrete 5-dimensional space containing ...
117 views

### Spatio-temopral wavelet analysis

Am quite new to wavelet analysis and would like some help. I am performing a spatio-temporal analysis of monthly gridded rainfall data. With PCA, I can reduce the dimension of the rainfall data into a ...
10k views

### Pandas limitations and its alternatives in Python

I read somewhere that Pandas was first developed for the financial world, at least not especially for natural sciences (physics, biology and so on), so is there any similar data analysis Python ...
353 views

### R/C/C++ library for N-dimensional arrays

I'm looking for a library that is either in R or easily wrappable with R, that can do the following things: construct and subset N-dimensional arrays perform operations such as ...
1k views

### Fit my data to Lissajous curve in Matlab

I'm acquiring data which looks like this: Theoretically it should be possible to fit it with a Lissajous curve, which is typically defined in the following way: $x = A \sin(a t + \delta)$, and \$y = ...
80 views

### Line scan in a 2D histogram

So I have a 2D histogram, in the third dimention are the counts. What I want to achieve is by choosing two random position, make a plot of a 1D histogram along the line that passes through the two ...