I had always assumed that the covariance matrix depends upon the amount and quality of your input data, but I am finding out that this is not the case. Is this true?

We want to fit $f(t) = \Sigma_{i=1}^{M} ( x_i t ) $ to the $N$ datapoints $(t_i,y_i)$ by using the matrix $A_{i,j} = \phi_j(t_i)$ where $\phi_j$ are the basis functions. Then $A$ is $N$x$M$ so we find the best fit by $X = (A^T .A)^{-1}.(A^T.Y)$.

Now, here the variance covariance matrix is $V = (A^T .A)^{-1}$, but it does not contain any information about the input data trying to be fitted $\{y_i\}$, only the form of the fitting function.

Is this really the case? Regardless of the input data, the covariance between the fitting parameters will be the same?


It is possible the see the effect of the data on the covariance if we take into account $ \{ \sigma_i \}$, the standard deviation of observations $\{ y_i \}$, but where does this appear in your covariance matrix? Do you create $A$ as $A_{i,j} = \sigma_i * \phi_j (t_i) $?


I can't quite read your notation, but suppose we tried to find a set of parameters $x$ that give rise to predictions $y(x)=Ax$ by comparing with measurements $\bar y$. Then, if you try to find your parameters $x$ by solving $$ \min_x \frac 12 \|\bar y - y(x)\|^2 = \frac 12 \|\bar y - Ax\|^2 $$ then it is indeed true that your covariance matrix is $(A^TA)^{-1}$ and is independent of both the data $\bar y$ and the data uncertainty.

However, in general, what we do is weight the misfit functional to use $$ \frac 12 \sum_i \frac 1{\sigma_i} |\bar y_i - y_i(x)|^2 $$ instead (this can be shown to be the correct formulation from a statistical viewpoint where you try to maximize the posterior probability), where $\sigma_i$ is the uncertainty in the $i$th measurement, and the covariance matrix that corresponds to this formulation than depends on the uncertainty.

Further, if you had a nonlinear model, i.e. not just $y(x)=Ax$, then the covariance matrix would be $$ V = \left(\nabla y(x^\ast)^T \nabla y(x^\ast)\right)^{-1} $$ where you take the derivative at the solution $x^\ast$ of the minimization problem, and this makes the covariance matrix depend on the actual data as well because the solution $x^\ast$ of course depends on the data.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.