Is there any clear classification between different iterative methods?
What is the difference between
Newton-like iterative methods?
A few specific examples:
Quasi-Newton methods avoid computing second derivatives by using an approximation to the Hessian which is updated at each iteration by a low rank (rank one or rank two) update. This makes factoring the Hessian (or equivalently keeping the inverse in product form) easy to do computationally. There are also limited memory Quasi-Newton methods that keep track of only the most recent rank one updates.
The Gauss-Newton method for minimization of sums of squares takes advantage of the sum of squares structure of the problem to get an approximation to the Hessian that only involves first derivatives (the second order term is dropped.)
The Levenberg-Marquardt method is a particular approach to stabilizing the Gauss-Newton iteration by regularizing the linear system that is solved in each iteration (either by adding a regularizing term in the equations or by using a trust region method.)