I am trying to minimize an objective function with 4 parameters, e.g., $a,b,c,d$ using gradient descent. $a < 0.1$, while $0 <b,c,d < 10$.

I'm using a learning rate for all parameters on the order of $10^{-3}$. So the gradient updates are also of the same order. Therefore, $b, c,$ and $d$ change very slowly.

How can I scale all my parameter so they are influenced by the updates on or about the same order? Or should I use a different learning rate for each parameter to increase/decrease the gradient magnitude?

  • $\begingroup$ Is your objective function scaled? $\endgroup$ – nicoguaro May 1 '18 at 14:49
  • 2
    $\begingroup$ These parameters aren't terribly badly scaled as is. Have you tried simply using a larger stepsize/learning rate? $\endgroup$ – Brian Borchers May 1 '18 at 15:26

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.