0
$\begingroup$

I'm trying to use the gradient ascent method on a convex function like the multivariate-Normal density function with respect to its parameters (the original is a bit more complicated), something similar to maximum likelihood estimation.

$$a_{n+1}=a_n+\gamma_n\nabla F(a_n)$$

I'm wondering if for the gradient ascent method will misbehave if I use a constant value for the step size, i.e. $\gamma_n=\gamma$.

Also, if I wanted to use an adaptive step size, what would be the simplest, and fastest computationally-wise way to define it?

$\endgroup$
2
$\begingroup$

It's not hard to couple gradient ascent with a line search method to determine the step size on the fly. Otherwise, for a fixed step size $\gamma$, your algorithm might require too many steps to maximize $F$ if $\gamma$ is too small or it may not find the local maximum if $\gamma$ is too big.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.