I am solving a nonlinear least squares problem using Gauss Newton method. Due to the large dimension of the problem, I use the Hessian-free approach. As a linear solver I use either MINRES or CG. To speed up convergence, I began to figure out which preconditioners can be easily added. First, I decided to try the Fisher preconditioner that Martens talks about in his work: $$ M = \left[ diag \left( \sum_{i=1}^{D} \nabla f_{i}(\theta)\odot \nabla f_{i}(\theta)\right) + \lambda I )\right]^a $$
In the article I use $\alpha$ = 0.75, but in my case it only worked at 0.25. The optimization error began to decrease faster, but the linear solution error increased relative to identity. Then I came across an interesting work Preconditioning for Hessian-Free Optimization by Robert Seidl.
In it, I was interested in the SPAI Preconditioner, where, using multiplication, we pull out the elements with the greatest gradients. To begin with, I decided to pull out the real diagonal elements corresponding to the greatest magnitude of the gradients, and use them to calculate the preconditioner. Leave the remaining elements of the diagonal equal to 1.
If I take the entire deagonal, the convergence is good, as soon as I reduce the number of elements by at least 2%, it becomes worse than the identity preconditioner. And this is strange, I assumed that any addition of information to the preconditioner should not make it worse than the identity preconditioner. A similar situation if I make SPAI preconditioner like Robert Seidl. Any advice on this?
