# Nonlinear least squares resolution matrix

For a linear least squares problem, it is possible to define a resolution matrix, relating the estimated model parameters to the true model parameters. If we are solving a regularized problem, $$\min_x || y - A x ||^2 + x^T \Lambda x$$ then $$x_{est} = (A^TA + \Lambda)^{-1} A^T y$$ and if we assume that $$y = A x_{true}$$, then we have $$x_{est} = R x_{true}$$ with the resolution matrix $$R = (A^TA + \Lambda)^{-1} A^TA$$. Resolution matrices are often used to study biases introduced into least squares by a priori regularization (the $$\Lambda$$ matrix).

However, what about nonlinear least squares? Is it possible to define an analogous resolution operator to study this? Here we have a residual vector $$r = y - f(x)$$ where $$f(x)$$ is a nonlinear function of $$x$$. We can assume (as in the linear case) that $$y = f(x_{true})$$, but we cannot write down a closed form expression for $$x_{est}$$, nor can we write $$f(x) = Ax$$ for some matrix $$A$$.

Does anyone know what can be done here? I have not had luck finding literature on this topic.

If you assume that the difference $$x_{est}-x_{true}$$ is small, then you can do a Taylor expansion and everything becomes linear again with $$A=\nabla f$$.
To make this clearer, recall that $$y=f(x_{true})$$, and so what you are trying to do is solve the problem $$\min_x || f(x_{true}) - f(x) ||^2 + x^T \Lambda x.$$ If you assume that there is not too much noise in the system, then $$f(x_{true}) \approx f(x)$$ and consequently we can make a Taylor expansion: $$f(x) \approx f(x_{true}) + \nabla f(x_{true})^T(x-x_{true})$$. That is, to first order the minimization problem is as follows: $$\min_x || \nabla f(x_{true})^T(x_{true}-x) ||^2 + x^T \Lambda x.$$ If you denote $$A=\nabla f(x_{true})$$ and rename $$z=Ax_{true}$$, then you see the relationship to the original, linear problem you were considering and that the resolution matrix is the same as before if you define $$A$$ as I have just done.
I'll point out that you might object having to set $$A=\nabla f(x_{true})$$ because you don't know what $$x_{true}$$ is, and consequently can't compute $$A$$. If you don't like that, then you can alternatively do the Taylor expansion the other way around: $$f(x_{true}) \approx f(x) + \nabla f(x)^T(x_{true}-x)$$. Going through the same formalism as above, you will see that you could just as well have set $$A=\nabla f(x_{est})$$ -- this is true to the same order in the Taylor expansion.
• Can you elaborate? With nonlinear least squares you never solve directly for $x_{est}$, you only solve for $\Delta x$ steps to bring you closer to $x_{est}$. So I don't see how you end up with a matrix relating $x_{est}$ and $x_{true}$. – vibe Feb 5 '20 at 4:34
• @vibe: How about now? The resolution matrix has nothing to do with how exactly you compute $x_{est}$, it is just a property of $x_{est}$ independent of how you obtain it. – Wolfgang Bangerth Feb 5 '20 at 13:21