Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Join us in building a kind, collaborative learning community via our updated Code of Conduct.

Data Analysis is the process of evaluating data using analytical and logical reasoning to examine each component of the data provided.

2
votes
0answers
24 views

Identifying the time window at which a meteorological factor influences tree growth the most

This is a problem I am encountering in a research project. I have thought of a naive approach to solve it which relies on many assumptions, is not guaranteed to work, and would be tedious and ...
0
votes
0answers
17 views

Measuring data quality metrics for streaming data

I need to research data quality and quantitative metrics for streaming data(for e.g. timeliness) in scientific literatures. I also need to implement the mechanism to capture the relevant metrics. The ...
0
votes
0answers
56 views

Finding correlations between variables in large SQLite database

I have an SQLite3 database file of ~150GB on an external hard-drive, and I would like to analyse correlations (if they exist) between columns of that database. To be precise, I'm interested in ...
0
votes
0answers
15 views

R imputation methods that preserves residuals and randomness

I am trying to use R package to impute missing data for stock prices. The challenge for common data imputation method is that, for usual data imputation methods, it generally tries to impute the ...
1
vote
0answers
16 views

Validating a Markov chain and Bagging to find overall model

let´s say I estimated a Markov chain by splitting my data into traning and test data, and estimating the transition matrix for the traning data yields $$ P= \begin{bmatrix} p11 & p12 &...
0
votes
1answer
93 views

How much in depth should I cover Python? [closed]

I'm an undergrad who's studying electronics and fields like data sciences and mostly computing seem most interesting to me but have in-depth knowledge in neither. I learnt basic python last winter ...
1
vote
0answers
29 views

Implementing Frechet Distance on pandas data frame [duplicate]

I am working on a trajectory analysis project using python and its data science related libraries. I am planning to implement Frechet Distance algorithm for trajectory analysis, each trajectory has ...
0
votes
2answers
876 views

Python implementation for Frechet Distance

I am working on a trajectory analysis project using python and its data science related libraries. I am planning to implement Frechet Distance algorithm for trajectory analysis, each trajectory has ...
1
vote
0answers
25 views

Use custom defined metric(not matrix) in Javaplex

I'm about to get started trying to use Javaplex for the first time for some application in topological data analysis (TDA) and I want to use a custom defined metric. From looking over the documents, I ...
8
votes
1answer
2k views

What is the difference between Computational Science and Data Science? [closed]

Background: My PhD was in 'Computational Science'. My dissertation was on the analysis of X-Ray Diffraction Data and analysis of thermally perturbed nuclei in the overall dynamic analysis of the ...
0
votes
0answers
55 views

Cloud computing solution for scientific Calculation software [duplicate]

I am an engineering student and we use a software called ROOT. https://root.cern.ch/ It is used for big data analysis and general scientific calculations, kind of like MATLAB I guess. My problem is ...
1
vote
2answers
38 views

How are the outcomes that generated from different predictive models combined to get more accurate predictions?

The simple average is commonly used to combine the predictions out of different predictive models. Apart form the simple average, what are the other methods that can be used for combining the ...
1
vote
0answers
41 views

Where to go next with clustered C Elegans neuronal activity series [closed]

I'm doing some independent research with the C Elegans nervous system (with the OpenWorm project) and was looking for some guidance as to where I should go next. Right now, I'm dealing with calcium ...
1
vote
1answer
257 views

FFT (Numerical Recipes in FORTRAN 77)

I am trying to use the subroutine twofft which in turn uses subroutine four1 in the book(Numerical Recipes in Fortran) to ...
6
votes
0answers
151 views

Is resampling more accurate than block average for statistical analysis of data?

I'm working in laboratories where molecular dynamics data are almost always analysed usign block average as stated in the famous Allen and Tildesley book. We divide the datas in blocks of size $M$ on ...
4
votes
1answer
321 views

Similarity of two CSV files (or more?)

A CSV file is characterized by a header, describing the n datarows to come. The header is a text string separated by a separator. A CSV header might look like (C1) ...
1
vote
1answer
59 views

library for arithmetic operations on unstructured xyz

I've got several sets of xyz-data where x,y are coordinates and z is a 'value'. I want to be able to do simple arithmetic operations on these data. The x,y locations are not necessarily the same in ...
3
votes
1answer
340 views

what is the best theory/model to use for prediction in multivariate data?

I use software for pollutant propagation on rivers that takes as input a set of parameters ($p_1,p_2,\ldots,p_n$) and creates an output file which is basically a matrix where on each row the ...
1
vote
1answer
68 views

Finite difference for Lighthill source term

I have experimental data containing horizontal and vertical components of speed and I need to evaluate this: $$ \frac{\partial^2}{\partial x_i \partial x_j}\left(v_iv_j\right) $$ Denoting $U$ as ...
2
votes
1answer
402 views

Is there a relationship between the covariance matrix and the partial derivative?

Suppose that there are $N$ pieces of data, each of which contains $M$ parameter values such that $N >> M$. If we put this information into matrix form ($N$ rows, $M$ columns) and then compute ...
2
votes
0answers
66 views

similarity/distance measurement between two ranked sequence

Is there an efficient way to measure similarity/distance between two sequences of ranked numbers/letters. The two sequences are of different length, and only have some elements in common? For example,...
1
vote
0answers
42 views

Manipulating/Extracting Data and Developing Methods - Language Choice [closed]

As a general programming enthusiast and aspiring Bioinformatician student I have an intermediate understanding of computing (languages) as well as Java, and to a lesser extent C++. Having knowledge in ...
5
votes
1answer
206 views

Iterative Cunnigham correction in Millikans oil-drop experiment

I tried to write an analysis program in python for the Millikan experiment based on a paper from Jones "The Millikan oil‐drop experiment: Making it worthwhile" However, I cannot reproduce the ...
0
votes
1answer
426 views

Power series regression linear fit in VBA excel

I wrote a program that calculates the best fit in VBA excel for the following model $$ y_k=c_1x_k+c_0+c_{-1}(x_k)^{-1} $$ solving for the best fit parameters $c_1$, $c_0$, and $c_{-1}$. However I ...
2
votes
1answer
39 views

Simple Path to Route Algorithm

I'm currently conducting research into subway paths, but stand in front of a problem, with no programming knowledge and an extremely short time-limit. Probably the question I am asking is trivial, ...
0
votes
1answer
77 views

Can Box-Cox transformation be applied for data of this form?

I have data of the form: X Y 3.53 0 4.93 50 5.53 60 6.21 70 7.37 80 9.98 90 16.56 100 And I want to find out $n$ so that this can be ...
0
votes
1answer
817 views

Gonzalez algorithm

I'm getting confused about implementing the Gonzalez algorithm for k-center clustering. The algorithm is : let $S$ be a data set Choose $z_1 \in S$ arbitrarily. Number $z$ is center. (In general ...
1
vote
0answers
35 views

Concatenating small windows of EEG signals over trial

My question is regarding the mathematical implications of concatenating very small windows from different trials, of the same size and for the same period in time after a stimuli. In other words, ...
0
votes
2answers
688 views

What is 'SOLVER' in R and Statistics/Analytics ?

**strong text**a) I tried to research on what exactly is a SOLVER only to find a not clear-cut simple answer. My doubts still remain after going through several sites full of discussions about it. I ...
5
votes
2answers
272 views

Dimensionality reduction of the domain of f(x)

I'm wondering if there is something analogous to a PCA for data sets where there is a dependent variable. (Though I am interested in any method of dimensionality reduction, PCA is just an example.) ...
3
votes
1answer
122 views

Monte Carlo Simulation - Random Number Motivation

For Monte Carlo simulations, or any other numerical methods that rely heavily on the quality of the pseudo-random numbers generated (i.e even/desired distribution on a certain domain) for that matter, ...
8
votes
4answers
459 views

Should I use a database to handle large amounts of results?

Background: I am currently running a large amount parameter variation experiments. They are being run in Python 2.6+, using numpy. These experiments are going to take about 2 weeks to run. Roughly I ...
1
vote
0answers
138 views

SciDB vs OLAP cube [closed]

I am working on good retrieval mechanism for big data systems. I am of view that multidimensional structures could be of great use for aggregations and querying . Multidimensional structures as ...
2
votes
2answers
234 views

Extracting time scales information from empirical cumulative distribution function

I have a stochastic process (a Markov chain actually) that has two absorbing states. I am using a difference equation to calculate the first passage time to either of the absorbing states. There are ...
0
votes
1answer
63 views

Finding maximum value in huge dataset

I am working on a project involving some pattern recognition. For this I need to find the maximum value in a huge multidimensional dataset. For example I have a discrete 5-dimensional space containing ...
3
votes
0answers
108 views

Spatio-temopral wavelet analysis

Am quite new to wavelet analysis and would like some help. I am performing a spatio-temporal analysis of monthly gridded rainfall data. With PCA, I can reduce the dimension of the rainfall data into a ...
9
votes
2answers
8k views

Pandas limitations and its alternatives in Python

I read somewhere that Pandas was first developed for the financial world, at least not especially for natural sciences (physics, biology and so on), so is there any similar data analysis Python ...
2
votes
3answers
291 views

R/C/C++ library for N-dimensional arrays

I'm looking for a library that is either in R or easily wrappable with R, that can do the following things: construct and subset N-dimensional arrays perform operations such as ...
6
votes
1answer
738 views

Fit my data to Lissajous curve in Matlab

I'm acquiring data which looks like this: Theoretically it should be possible to fit it with a Lissajous curve, which is typically defined in the following way: $x = A \sin(a t + \delta)$, and $y = ...
1
vote
1answer
63 views

Line scan in a 2D histogram

So I have a 2D histogram, in the third dimention are the counts. What I want to achieve is by choosing two random position, make a plot of a 1D histogram along the line that passes through the two ...
1
vote
0answers
38 views

Spectra of the energy flux VS transfer spectra?

In direct numerical simulation of turbulence what is the advantage to look at the spectra of the energy flux rather than the transfer spectra? The transfer spectra corresponds to the triple-velocity ...
1
vote
1answer
36 views

Adding data with different abscissas

This question may be better suited for an Astronomy Stack Exchange site, but I figured I'd ask here. Say I have measurements of something as a function of radius for a number of objects. Here's an ...
3
votes
2answers
66 views

Parameter Fitting: Need measure of data 'support' for a parameter solution

I am estimating parameters on a dataset that would, for the most part, result in a weakly constrained solution. The dataset however also contains a few more data points that make the solution well-...
1
vote
1answer
67 views

Extract clusters from a graph of absolute-distance edge

I don't know if I have formulated this problem right: I have tons of items and the distances between each pair of them. Feeding this data into some visualization tool, I am able to create a nice ...
2
votes
0answers
55 views

Error in treating non-uniformly spaced data as uniform

In discussing smoothing filters, Numerical Recipes p. 772 says ... irregularly sampled data, where the values $f_i$ are not uniformly spaced ... one can simply pretend that the data points are ...
3
votes
1answer
674 views

How to do local FFT on huge 3D vector data cell mesh and visualize it spatially?

Simulation type: I'm running a simulation with the OOMMF micromagnetics package http://math.nist.gov/oommf/ where are magnet is represented by a mesh of 3 million cells, it gets excited by a ...
1
vote
1answer
100 views

classification machinery needed

Consider a set of 7D vectors. Each vector belongs to one of four classes. After mapping to 3D with PCA and coloring each point according its class the dataset looks like as shown below: For the ...
1
vote
1answer
299 views

Clustering data based on the radius of a sphere

I have a data-set that has four columns [X Y Z C]. I would like to find all the C values that are in a given sphere centered at [X, Y, Z] with a radius r. What is the best approach to address this ...
5
votes
1answer
261 views

Recommendation for a python book for data processing

I've been doing FORTRAN programming for 10 years and I've started using python for a few years now, mostly for data processing. I've been lucky enough to work with people that are knowledgeable ...
2
votes
2answers
79 views

Estimate (non-)drift in noisy data

I have a time series representing the result of a complex calculation (physical simulation). Due to round-off errors and approximation errors, there will be some "noise" on the data series. In some ...