Nonlinear least squares resolution matrix

Question

For a linear least squares problem, it is possible to define a resolution matrix, relating the estimated model parameters to the true model parameters. If we are solving a regularized problem, $$ \min_x || y - A x ||^2 + x^T \Lambda x $$ then $$ x_{est} = (A^TA + \Lambda)^{-1} A^T y $$ and if we assume that $y = A x_{true}$, then we have $$ x_{est} = R x_{true} $$ with the resolution matrix $R = (A^TA + \Lambda)^{-1} A^TA$. Resolution matrices are often used to study biases introduced into least squares by a priori regularization (the $\Lambda$ matrix).

However, what about nonlinear least squares? Is it possible to define an analogous resolution operator to study this? Here we have a residual vector $$ r = y - f(x) $$ where $f(x)$ is a nonlinear function of $x$. We can assume (as in the linear case) that $y = f(x_{true})$, but we cannot write down a closed form expression for $x_{est}$, nor can we write $f(x) = Ax$ for some matrix $A$.

Does anyone know what can be done here? I have not had luck finding literature on this topic.

Answer 1

If you assume that the difference $x_{est}-x_{true}$ is small, then you can do a Taylor expansion and everything becomes linear again with $A=\nabla f$.

To make this clearer, recall that $y=f(x_{true})$, and so what you are trying to do is solve the problem $$ \min_x || f(x_{true}) - f(x) ||^2 + x^T \Lambda x. $$ If you assume that there is not too much noise in the system, then $f(x_{true}) \approx f(x)$ and consequently we can make a Taylor expansion: $f(x) \approx f(x_{true}) + \nabla f(x_{true})^T(x-x_{true})$. That is, to first order the minimization problem is as follows: $$ \min_x || \nabla f(x_{true})^T(x_{true}-x) ||^2 + x^T \Lambda x. $$ If you denote $A=\nabla f(x_{true})$ and rename $z=Ax_{true}$, then you see the relationship to the original, linear problem you were considering and that the resolution matrix is the same as before if you define $A$ as I have just done.

I'll point out that you might object having to set $A=\nabla f(x_{true})$ because you don't know what $x_{true}$ is, and consequently can't compute $A$. If you don't like that, then you can alternatively do the Taylor expansion the other way around: $f(x_{true}) \approx f(x) + \nabla f(x)^T(x_{true}-x)$. Going through the same formalism as above, you will see that you could just as well have set $A=\nabla f(x_{est})$ -- this is true to the same order in the Taylor expansion.