Return to Answer

Nicol N. Schraudolph spent a few years on Krylov-like methods for Machine Learning. I first learned of his work in 2004 Machine Learning Summer School in Canberra.

Personally I've spent significant time working on ways of improving SGD (contributing to tools like this), and came to believe that for optimized architectures like Resnet-50, optimally preconditioned direction will only give a modest improvement, perhaps $\approx$ 5x reduction in terms of the number of steps needed. This means if your advanced method does more than "5 SGD steps" worth of work per step, it'll lose to SGD. Additionally there's a storage constraint and you can't store more than a couple gradients worth of statistics (models are typically scaled up until gradient computation exceeds GPU memory constraint)

• overparameterized and very high-dimensional. SGD in this setting is already pretty good, converging at an exponential rate -- https://arxiv.org/abs/1811.02564
• non-stationary. Local curvature statistics get stale after a few of steps.
• not optimized to convergence. Many "advanced methods" are theoretically superior to SGD only in the "tail regime", but we don't get there
• tailor made for SGD. If SGD fails to achieve good results, corresponding network architecture won't see the light of day. For instance, autoencoder with a bottleneck layer is impossible to train with SGD, but easy with a factored Gauss-Newton method. Since advanced optimizers aren't accessible in ML frameworks, such architecture wouldn't get a shot at evaluation.

1. stores preconditioner $P$ in a factored form, in extreme case, parameterizing it by a single number recovers SGD with "automatic step size"
2. updates $P$ in an online fashion by taking cheap SGD steps on $P$ to keep $P^{-1}$ close to Hessian $H$.

Nicol N. Schraudolph spent a few years on Krylov-like methods for Machine Learning. I first learned of his work at 2004 Machine Learning Summer School in Canberra.

Personally I've spent significant time working on ways of improving SGD for ML (contributing to tools like this), and came to believe that for optimized architectures like Resnet-50, optimally preconditioned direction would only give a modest improvement, perhaps $\approx$ 5x reduction in terms of the number of steps needed. This means if your advanced method does more than "5 SGD steps" worth of work per step, it'll lose to SGD. Additionally there's a storage constraint which means you can't store more than a couple gradients worth of statistics throughout training. Models are typically scaled up until gradient computation exceeds GPU memory constraint.

• overparameterized and very high-dimensional. SGD in this setting is already pretty good, converging at an exponential rate -- https://arxiv.org/abs/1811.02564

• non-stationary. Local curvature statistics get stale after a few of steps.

• not optimized to convergence. Many "advanced methods" are theoretically superior to SGD only in the "tail regime", but we don't get there. For instance, in BigScience model training, the loss went down from $\approx 10$ to 0.13 in the last 12 months of training. So we care about the time it takes to decrease loss by a factor of 100. Meanwhile second-order methods like Newton method starts to pull away in the tail regime, going from $10^{-5}$ loss to $10^{-10}$.

• tailor made for SGD. If SGD fails to achieve good results, corresponding network architecture won't see the light of day. For instance, autoencoder with a bottleneck layer is near impossible to train with SGD, but easy with a factored Gauss-Newton method. Since advanced optimizers aren't accessible in ML frameworks, such architecture wouldn't get a shot at evaluation.

1. stores preconditioner $P$ in a factored form, in extreme case, parameterizing it by a single number recovers SGD with "automatic step size"
2. updates $P$ in an online fashion by taking cheap SGD steps on $P$ to keep $P^{-1}$ close to Hessian $H$, using Hessian-vector products.

(I had no hand in PSGD, other than having evaluated it and found it to be the most promising out of all second-order methods I've used. I think it didn't get attention like KFAC or Shampoo because it didn't come from a famous ML lab)

They aren't popular because they don't work.

Nicol N. Schraudolph spent a few years on Krylov-like methods for Machine Learning. I first learned of his work in 2004 Machine Learning Summer School in Canberra.

Looking at his later work (A Stochastic Quasi-Newton Method for Online Convex Optimization) you can find the following explanation:

More generally, thousands of people have tried to improve on SGD+momentum in Machine Learning setting, without success. Adam is slightly better on many tasks, but slightly worse on others, like Imagenet training.

Personally I've spent significant time working on ways of improving SGD (contributing to tools like this), and came to believe that for optimized architectures like Resnet-50, optimally preconditioned direction will only give a modest improvement, perhaps $\approx$ 5x reduction in terms of the number of steps needed. This means if your advanced method does more than "5 SGD steps" worth of work per step, it'll lose to SGD. Additionally there's a storage constraint and you can't store more than a couple gradients worth of statistics (models are typically scaled up until gradient computation exceeds GPU memory constraint)

General reasons why it's hard to improve on SGD in ML. Typical problems are:

• overparameterized and very high-dimensional. SGD in this setting is already pretty good, converging at an exponential rate -- https://arxiv.org/abs/1811.02564
• non-stationary. Local curvature statistics get stale after a few of steps.
• not optimized to convergence. Many "advanced methods" are theoretically superior to SGD only in the "tail regime", but we don't get there
• tailor made for SGD. If SGD fails to achieve good results, corresponding network architecture won't see the light of day. For instance, autoencoder with a bottleneck layer is impossible to train with SGD, but easy with a factored Gauss-Newton method. Since advanced optimizers aren't accessible in ML frameworks, such architecture wouldn't get a shot at evaluation.

An example of method which satisfies these constraints is PSGD which:

1. stores preconditioner $P$ in a factored form
2. updates $P$ in an online fashion by taking cheap SGD steps on $P$ to keep $\|I-PH\|$ small

They aren't popular because they don't work.

Nicol N. Schraudolph spent a few years on Krylov-like methods for Machine Learning. I first learned of his work in 2004 Machine Learning Summer School in Canberra.

Looking at his later work (A Stochastic Quasi-Newton Method for Online Convex Optimization) you can find the following explanation:

More generally, thousands of people have tried to improve on SGD+momentum in Machine Learning setting, without success. Adam is slightly better on many tasks, but slightly worse on others, like Imagenet training.

Personally I've spent significant time working on ways of improving SGD (contributing to tools like this), and came to believe that for optimized architectures like Resnet-50, optimally preconditioned direction will only give a modest improvement, perhaps $\approx$ 5x reduction in terms of the number of steps needed. This means if your advanced method does more than "5 SGD steps" worth of work per step, it'll lose to SGD. Additionally there's a storage constraint and you can't store more than a couple gradients worth of statistics (models are typically scaled up until gradient computation exceeds GPU memory constraint)

General reasons why it's hard to improve on SGD in ML. Typical problems are:

• overparameterized and very high-dimensional. SGD in this setting is already pretty good, converging at an exponential rate -- https://arxiv.org/abs/1811.02564
• non-stationary. Local curvature statistics get stale after a few of steps.
• not optimized to convergence. Many "advanced methods" are theoretically superior to SGD only in the "tail regime", but we don't get there
• tailor made for SGD. If SGD fails to achieve good results, corresponding network architecture won't see the light of day. For instance, autoencoder with a bottleneck layer is impossible to train with SGD, but easy with a factored Gauss-Newton method. Since advanced optimizers aren't accessible in ML frameworks, such architecture wouldn't get a shot at evaluation.

An example of method which satisfies these constraints is PSGD which:

1. stores preconditioner $P$ in a factored form, in extreme case, parameterizing it by a single number recovers SGD with "automatic step size"
2. updates $P$ in an online fashion by taking cheap SGD steps on $P$ to keep $P^{-1}$ close to Hessian $H$.

They aren't popular because they don't work.

Nicol N. Schraudolph spent a few years trying to make Krylov methods work for Machine Learning. I first learned of his work in 2004 Machine Learning Summer School in Canberra.

Looking at his later work (A Stochastic Quasi-Newton Method for Online Convex Optimization) you can find the following explanation:

More generally, thousands of people have tried to improve on SGD+momentum in Machine Learning setting, without success. Adam is slightly better on many tasks, but slightly worse on others, like Imagenet training.

Personally I've spent significant time working on ways of improving SGD (contributing to tools like this), and believe that for optimized architectures like Resnet-50, optimally preconditioned direction will only give a modest improvement, perhaps $\approx$ 5x reduction in terms of the number of steps needed. This means if your advanced method does more than "5 SGD steps" worth of work per step, it'll lose to SGD. Additionally there's a storage constraint and you can't store more than a few gradients worth of statistics (models are typically scaled up until gradient computation exceeds GPU memory constraint)

General reasons why it's hard to improve on SGD in ML. Typical problems are:

• overparameterized and very high-dimensional. SGD in this setting is already pretty good, converging at an exponential rate -- https://arxiv.org/abs/1811.02564
• non-stationary. Local curvature statistics get stale after a few of steps.
• not optimized to convergence. Many "advanced methods" are theoretically superior to SGD only in the "tail regime", but we don't get there
• tailor made for SGD. If SGD fails to achieve good results, corresponding network architecture won't see the light of day. For instance, autoencoder with a bottleneck layer is impossible to train with SGD, but easy with a factored Gauss-Newton method. Since advanced optimizers aren't accessible in ML frameworks, such architecture wouldn't get a shot at evaluation.

An example of method which satisfies these constraints is PSGD which:

1. stores preconditioner $P$ in a factored form
2. updates $P$ in an online fashion by taking cheap SGD steps on $P$ to keep $\|I-PH\|$ small

They aren't popular because they don't work.

Nicol N. Schraudolph spent a few years on Krylov-like methods for Machine Learning. I first learned of his work in 2004 Machine Learning Summer School in Canberra.

Looking at his later work (A Stochastic Quasi-Newton Method for Online Convex Optimization) you can find the following explanation:

More generally, thousands of people have tried to improve on SGD+momentum in Machine Learning setting, without success. Adam is slightly better on many tasks, but slightly worse on others, like Imagenet training.

Personally I've spent significant time working on ways of improving SGD (contributing to tools like this), and came to believe that for optimized architectures like Resnet-50, optimally preconditioned direction will only give a modest improvement, perhaps $\approx$ 5x reduction in terms of the number of steps needed. This means if your advanced method does more than "5 SGD steps" worth of work per step, it'll lose to SGD. Additionally there's a storage constraint and you can't store more than a couple gradients worth of statistics (models are typically scaled up until gradient computation exceeds GPU memory constraint)

General reasons why it's hard to improve on SGD in ML. Typical problems are:

• overparameterized and very high-dimensional. SGD in this setting is already pretty good, converging at an exponential rate -- https://arxiv.org/abs/1811.02564
• non-stationary. Local curvature statistics get stale after a few of steps.
• not optimized to convergence. Many "advanced methods" are theoretically superior to SGD only in the "tail regime", but we don't get there
• tailor made for SGD. If SGD fails to achieve good results, corresponding network architecture won't see the light of day. For instance, autoencoder with a bottleneck layer is impossible to train with SGD, but easy with a factored Gauss-Newton method. Since advanced optimizers aren't accessible in ML frameworks, such architecture wouldn't get a shot at evaluation.

An example of method which satisfies these constraints is PSGD which:

1. stores preconditioner $P$ in a factored form
2. updates $P$ in an online fashion by taking cheap SGD steps on $P$ to keep $\|I-PH\|$ small