# Does the covariance matrix in Least Squares depend upon the input data?

I had always assumed that the covariance matrix depends upon the amount and quality of your input data, but I am finding out that this is not the case. Is this true?

We want to fit $f(t) = \Sigma_{i=1}^{M} ( x_i t )$ to the $N$ datapoints $(t_i,y_i)$ by using the matrix $A_{i,j} = \phi_j(t_i)$ where $\phi_j$ are the basis functions. Then $A$ is $N$x$M$ so we find the best fit by $X = (A^T .A)^{-1}.(A^T.Y)$.

Now, here the variance covariance matrix is $V = (A^T .A)^{-1}$, but it does not contain any information about the input data trying to be fitted $\{y_i\}$, only the form of the fitting function.

Is this really the case? Regardless of the input data, the covariance between the fitting parameters will be the same?

EDIT:

It is possible the see the effect of the data on the covariance if we take into account $\{ \sigma_i \}$, the standard deviation of observations $\{ y_i \}$, but where does this appear in your covariance matrix? Do you create $A$ as $A_{i,j} = \sigma_i * \phi_j (t_i)$?

I can't quite read your notation, but suppose we tried to find a set of parameters $x$ that give rise to predictions $y(x)=Ax$ by comparing with measurements $\bar y$. Then, if you try to find your parameters $x$ by solving $$\min_x \frac 12 \|\bar y - y(x)\|^2 = \frac 12 \|\bar y - Ax\|^2$$ then it is indeed true that your covariance matrix is $(A^TA)^{-1}$ and is independent of both the data $\bar y$ and the data uncertainty.
However, in general, what we do is weight the misfit functional to use $$\frac 12 \sum_i \frac 1{\sigma_i} |\bar y_i - y_i(x)|^2$$ instead (this can be shown to be the correct formulation from a statistical viewpoint where you try to maximize the posterior probability), where $\sigma_i$ is the uncertainty in the $i$th measurement, and the covariance matrix that corresponds to this formulation than depends on the uncertainty.
Further, if you had a nonlinear model, i.e. not just $y(x)=Ax$, then the covariance matrix would be $$V = \left(\nabla y(x^\ast)^T \nabla y(x^\ast)\right)^{-1}$$ where you take the derivative at the solution $x^\ast$ of the minimization problem, and this makes the covariance matrix depend on the actual data as well because the solution $x^\ast$ of course depends on the data.