I have a time series representing the result of a complex calculation (physical simulation). Due to round-off errors and approximation errors, there will be some "noise" on the data series. In some cases, the physical simulation represents a "steady state" situation meaning that, in theory, the time series should be a flat line. However, if the steady state is not well calculated there will be a drift in the time series.

What would be the best algorithm to check whether there is drift or not? Some smoothing filter followed by a kind of linear function approximation $ax+b$ and check for a small $a$ (define small?)


2 Answers 2


I'd follow your suggestion of fitting a line $ax+b$ to your data. You don't even have to smooth it first -- fitting a line already takes care of it.

There are, however, two questions: first, how do you fit the line? This is a question of the statistics of your noise. If noise is Gaussian then you'd use least squares. If your noise has large outliers then you'd want to use a L1-minimization (least absolute values) to fit $a,b$.

The second part is to determine whether your drift is significant. To this end, after fitting your predicted drift is $d=a(x_1-x_0)$ if your $x$ values are in the interval $[x_0,x_1]$. $d$ is significant if it is larger than, say, one standard deviation $\sigma$ of your noise where you can compute $\sigma$ as $\sigma=\sqrt{\frac 1N \sum_i (y_i-(ax_i+b))^2}$. If $d\gg \sigma$ you can be sure that there is drift.

  • $\begingroup$ In the presence of noise and outliers, an l1 fit is not quite the right approach, but one should use robust regressein with a huber functional. $\endgroup$ Commented Oct 24, 2012 at 10:48
  • $\begingroup$ Well, I'd say the form of the minimization functional is given by the statistical properties of the noise. $l_1$ minimization is appropriate if noise is distributed as $e^{-|\varepsilon|/\sigma}$. $\endgroup$ Commented Oct 24, 2012 at 10:51
  • $\begingroup$ Yes, but this is untypical for noise, and usually accounts only for the outlier part. $\endgroup$ Commented Oct 24, 2012 at 10:52
  • $\begingroup$ I don't know. I guess it depends on the measurement device. As a funny aside, we always do least squares because we claim that our error is Gaussian. I was absolutely certain that that's an assumption that is never true in practice (why would it?) but in a class I taught on parameter estimation, we looked at 100,000 exposures of a camera of the same scene and analyzed the noise -- and lo and behold, it is almost perfectly fit by a Gaussian. Who would have guessed :-) What I meant to say is simply that one can't tell unless one measures the noise characteristics. $\endgroup$ Commented Oct 24, 2012 at 12:45
  • $\begingroup$ The central limit theorem gives a good reason why certain applications lead to Gaussian noise. In others the noise can be very far from Gaussian. - However, linear least squares is optimal for any uncorrelated noise with constant variance, and needs no Gaussian assumption. $\endgroup$ Commented Oct 24, 2012 at 14:32

You should try a Kalman filter, if possible for your problem. It's used in accelerometers, which also tend to drift.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.