For a linear least squares problem, it is possible to define a resolution matrix, relating the estimated model parameters to the true model parameters. If we are solving a regularized problem, $$ \min_x || y - A x ||^2 + x^T \Lambda x $$ then $$ x_{est} = (A^TA + \Lambda)^{-1} A^T y $$ and if we assume that $y = A x_{true}$, then we have $$ x_{est} = R x_{true} $$ with the resolution matrix $R = (A^TA + \Lambda)^{-1} A^TA$. Resolution matrices are often used to study biases introduced into least squares by a priori regularization (the $\Lambda$ matrix).

However, what about nonlinear least squares? Is it possible to define an analogous resolution operator to study this? Here we have a residual vector $$ r = y - f(x) $$ where $f(x)$ is a nonlinear function of $x$. We can assume (as in the linear case) that $y = f(x_{true})$, but we cannot write down a closed form expression for $x_{est}$, nor can we write $f(x) = Ax$ for some matrix $A$.

Does anyone know what can be done here? I have not had luck finding literature on this topic.


1 Answer 1


If you assume that the difference $x_{est}-x_{true}$ is small, then you can do a Taylor expansion and everything becomes linear again with $A=\nabla f$.

To make this clearer, recall that $y=f(x_{true})$, and so what you are trying to do is solve the problem $$ \min_x || f(x_{true}) - f(x) ||^2 + x^T \Lambda x. $$ If you assume that there is not too much noise in the system, then $f(x_{true}) \approx f(x)$ and consequently we can make a Taylor expansion: $f(x) \approx f(x_{true}) + \nabla f(x_{true})^T(x-x_{true})$. That is, to first order the minimization problem is as follows: $$ \min_x || \nabla f(x_{true})^T(x_{true}-x) ||^2 + x^T \Lambda x. $$ If you denote $A=\nabla f(x_{true})$ and rename $z=Ax_{true}$, then you see the relationship to the original, linear problem you were considering and that the resolution matrix is the same as before if you define $A$ as I have just done.

I'll point out that you might object having to set $A=\nabla f(x_{true})$ because you don't know what $x_{true}$ is, and consequently can't compute $A$. If you don't like that, then you can alternatively do the Taylor expansion the other way around: $f(x_{true}) \approx f(x) + \nabla f(x)^T(x_{true}-x)$. Going through the same formalism as above, you will see that you could just as well have set $A=\nabla f(x_{est})$ -- this is true to the same order in the Taylor expansion.

  • $\begingroup$ Can you elaborate? With nonlinear least squares you never solve directly for $x_{est}$, you only solve for $\Delta x$ steps to bring you closer to $x_{est}$. So I don't see how you end up with a matrix relating $x_{est}$ and $x_{true}$. $\endgroup$
    – vibe
    Feb 5, 2020 at 4:34
  • $\begingroup$ @vibe: How about now? The resolution matrix has nothing to do with how exactly you compute $x_{est}$, it is just a property of $x_{est}$ independent of how you obtain it. $\endgroup$ Feb 5, 2020 at 13:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.