# Methods for Parameter Scaling in Gradient-based Optimization

I am trying to minimize an objective function with 4 parameters, e.g., $a,b,c,d$ using gradient descent. $a < 0.1$, while $0 <b,c,d < 10$.

I'm using a learning rate for all parameters on the order of $10^{-3}$. So the gradient updates are also of the same order. Therefore, $b, c,$ and $d$ change very slowly.

How can I scale all my parameter so they are influenced by the updates on or about the same order? Or should I use a different learning rate for each parameter to increase/decrease the gradient magnitude?

• Is your objective function scaled? – nicoguaro May 1 '18 at 14:49
• These parameters aren't terribly badly scaled as is. Have you tried simply using a larger stepsize/learning rate? – Brian Borchers May 1 '18 at 15:26