Tell me more ×
Computational Science Stack Exchange is a question and answer site for scientists using computers to solve scientific problems. It's 100% free, no registration required.
up vote 1 down vote favorite

I have a time series representing the result of a complex calculation (physical simulation). Due to round-off errors and approximation errors, there will be some "noise" on the data series. In some cases, the physical simulation represents a "steady state" situation meaning that, in theory, the time series should be a flat line. However, if the steady state is not well calculated there will be a drift in the time series.

What would be the best algorithm to check whether there is drift or not? Some smoothing filter followed by a kind of linear function approximation $ax+b$ and check for a small $a$ (define small?)

share|improve this question

2 Answers

up vote 3 down vote accepted

I'd follow your suggestion of fitting a line $ax+b$ to your data. You don't even have to smooth it first -- fitting a line already takes care of it.

There are, however, two questions: first, how do you fit the line? This is a question of the statistics of your noise. If noise is Gaussian then you'd use least squares. If your noise has large outliers then you'd want to use a L1-minimization (least absolute values) to fit $a,b$.

The second part is to determine whether your drift is significant. To this end, after fitting your predicted drift is $d=a(x_1-x_0)$ if your $x$ values are in the interval $[x_0,x_1]$. $d$ is significant if it is larger than, say, one standard deviation $\sigma$ of your noise where you can compute $\sigma$ as $\sigma=\sqrt{\frac 1N \sum_i (y_i-(ax_i+b))^2}$. If $d\gg \sigma$ you can be sure that there is drift.

share|improve this answer
In the presence of noise and outliers, an l1 fit is not quite the right approach, but one should use robust regressein with a huber functional. – Arnold Neumaier Oct 24 '12 at 10:48
Well, I'd say the form of the minimization functional is given by the statistical properties of the noise. $l_1$ minimization is appropriate if noise is distributed as $e^{-|\varepsilon|/\sigma}$. – Wolfgang Bangerth Oct 24 '12 at 10:51
Yes, but this is untypical for noise, and usually accounts only for the outlier part. – Arnold Neumaier Oct 24 '12 at 10:52
I don't know. I guess it depends on the measurement device. As a funny aside, we always do least squares because we claim that our error is Gaussian. I was absolutely certain that that's an assumption that is never true in practice (why would it?) but in a class I taught on parameter estimation, we looked at 100,000 exposures of a camera of the same scene and analyzed the noise -- and lo and behold, it is almost perfectly fit by a Gaussian. Who would have guessed :-) What I meant to say is simply that one can't tell unless one measures the noise characteristics. – Wolfgang Bangerth Oct 24 '12 at 12:45
The central limit theorem gives a good reason why certain applications lead to Gaussian noise. In others the noise can be very far from Gaussian. - However, linear least squares is optimal for any uncorrelated noise with constant variance, and needs no Gaussian assumption. – Arnold Neumaier Oct 24 '12 at 14:32
show 1 more comment
up vote 0 down vote

You should try a Kalman filter, if possible for your problem. It's used in accelerometers, which also tend to drift.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.