They aren't popular because they don't work.
Nicol N. Schraudolph spent a few years on Krylov-like methods for Machine Learning. I first learned of his work in 2004 Machine Learning Summer School in Canberra.
Looking at his later work (A Stochastic Quasi-Newton Method for Online Convex Optimization) you can find the following explanation:
More generally, thousands of people have tried to improve on SGD+momentum in Machine Learning setting, without success. Adam is slightly better on many tasks, but slightly worse on others, like Imagenet training.
Personally I've spent significant time working on ways of improving SGD (contributing to tools like this), and came to believe that for optimized architectures like Resnet-50, optimally preconditioned direction will only give a modest improvement, perhaps $\approx$ 5x reduction in terms of the number of steps needed. This means if your advanced method does more than "5 SGD steps" worth of work per step, it'll lose to SGD. Additionally there's a storage constraint and you can't store more than a couple gradients worth of statistics (models are typically scaled up until gradient computation exceeds GPU memory constraint)
General reasons why it's hard to improve on SGD in ML. Typical problems are:
- overparameterized and very high-dimensional. SGD in this setting is already pretty good, converging at an exponential rate -- https://arxiv.org/abs/1811.02564
- non-stationary. Local curvature statistics get stale after a few of steps.
- not optimized to convergence. Many "advanced methods" are theoretically superior to SGD only in the "tail regime", but we don't get there
- tailor made for SGD. If SGD fails to achieve good results, corresponding network architecture won't see the light of day. For instance, autoencoder with a bottleneck layer is impossible to train with SGD, but easy with a factored Gauss-Newton method. Since advanced optimizers aren't accessible in ML frameworks, such architecture wouldn't get a shot at evaluation.
An example of method which satisfies these constraints is PSGD which:
- stores preconditioner $P$ in a factored form
- updates $P$ in an online fashion by taking cheap SGD steps on $P$ to keep $\|I-PH\|$ small