I am interested in the specific differences of the following methods:
- The conjugate gradient method (CGM) is an algorithm for the numerical solution of particular systems of linear equations.
- The nonlinear conjugate gradient method (NLCGM) generalizes the conjugate gradient method to nonlinear optimization.
- The gradient descent/steepest descent algorithm (GDA) is a first-order iterative optimization algorithm.
- The stochastic gradient descent (SGD) is a stochastic approximation of the gradient descent optimization method for minimizing an objective function that is written as a sum of differentiable functions.
Why do some publications introduce the CGM as a method for the solution of systems of equations, when it would be much more intuitive to introduce it as a linear optimization algorithm, for which one can define the residual of the system as the objective function, to then portrait the solution of a linear system, reformulated as a quadratic minimization problem?
I know that the GDA can also be used to solve linear system of equations but it is rarely used, since the conjugate gradient method is one of the most popular alternatives. However, gradient descent can also be used to solve a system of nonlinear equations. But, besides conjugacy, convergence and benefits from preconditioning, are there any other differences worth mentioning?
To add some context to this question: I am trying to better understand the different first order optimization algorithms, in order to make a more efficient use of them in inversion problems/deep learning problems.