More specifically, when training a neural network, what reasons are there for choosing an optimizer from the family consisting of stochastic gradient descent (SGD) and its extensions (RMSProp, Adam, etc.) instead of from the family of Quasi-Newton methods (including limited-memory BFGS, abbreviated as L-BFGS)?
It is clear to me that some of the extensions of SGD, particularly RMSProp and Adam, store gradient information from previous iterates and use it to calculate the update for the next iterate. This is something they have in common with Quasi-Newton methods. However, the motivation behind storing the gradient information in the Adam method, for example, is unclear to me, whereas it is clear that in Quasi-Newton methods the motivation behind storing prior gradient information is to use it to construct an approximation to the Hessian (inverse).
I'm curious to understand:
- whether there are features of methods like Adam that make them particularly well-suited to machine learning applications;
- whether these features enable them to outperform more conventional optimization methods like Quasi-Newton methods; and (ideally)
- what is the general reasoning behind the update rule that these methods use.