Estimate (non-)drift in noisy data

I have a time series representing the result of a complex calculation (physical simulation). Due to round-off errors and approximation errors, there will be some "noise" on the data series. In some cases, the physical simulation represents a "steady state" situation meaning that, in theory, the time series should be a flat line. However, if the steady state is not well calculated there will be a drift in the time series.

What would be the best algorithm to check whether there is drift or not? Some smoothing filter followed by a kind of linear function approximation $ax+b$ and check for a small $a$ (define small?)

-

2 Answers

I'd follow your suggestion of fitting a line $ax+b$ to your data. You don't even have to smooth it first -- fitting a line already takes care of it.

There are, however, two questions: first, how do you fit the line? This is a question of the statistics of your noise. If noise is Gaussian then you'd use least squares. If your noise has large outliers then you'd want to use a L1-minimization (least absolute values) to fit $a,b$.

The second part is to determine whether your drift is significant. To this end, after fitting your predicted drift is $d=a(x_1-x_0)$ if your $x$ values are in the interval $[x_0,x_1]$. $d$ is significant if it is larger than, say, one standard deviation $\sigma$ of your noise where you can compute $\sigma$ as $\sigma=\sqrt{\frac 1N \sum_i (y_i-(ax_i+b))^2}$. If $d\gg \sigma$ you can be sure that there is drift.

-
In the presence of noise and outliers, an l1 fit is not quite the right approach, but one should use robust regressein with a huber functional. – Arnold Neumaier Oct 24 '12 at 10:48
Well, I'd say the form of the minimization functional is given by the statistical properties of the noise. $l_1$ minimization is appropriate if noise is distributed as $e^{-|\varepsilon|/\sigma}$. – Wolfgang Bangerth Oct 24 '12 at 10:51
Yes, but this is untypical for noise, and usually accounts only for the outlier part. – Arnold Neumaier Oct 24 '12 at 10:52
I don't know. I guess it depends on the measurement device. As a funny aside, we always do least squares because we claim that our error is Gaussian. I was absolutely certain that that's an assumption that is never true in practice (why would it?) but in a class I taught on parameter estimation, we looked at 100,000 exposures of a camera of the same scene and analyzed the noise -- and lo and behold, it is almost perfectly fit by a Gaussian. Who would have guessed :-) What I meant to say is simply that one can't tell unless one measures the noise characteristics. – Wolfgang Bangerth Oct 24 '12 at 12:45
The central limit theorem gives a good reason why certain applications lead to Gaussian noise. In others the noise can be very far from Gaussian. - However, linear least squares is optimal for any uncorrelated noise with constant variance, and needs no Gaussian assumption. – Arnold Neumaier Oct 24 '12 at 14:32

You should try a Kalman filter, if possible for your problem. It's used in accelerometers, which also tend to drift.

-