Gradient ascent method with a constant step size?

Question

I'm trying to use the gradient ascent method on a convex function like the multivariate-Normal density function with respect to its parameters (the original is a bit more complicated), something similar to maximum likelihood estimation.

$$a_{n+1}=a_n+\gamma_n\nabla F(a_n)$$

I'm wondering if for the gradient ascent method will misbehave if I use a constant value for the step size, i.e. $\gamma_n=\gamma$.

Also, if I wanted to use an adaptive step size, what would be the simplest, and fastest computationally-wise way to define it?

Answer 1

It's not hard to couple gradient ascent with a line search method to determine the step size on the fly. Otherwise, for a fixed step size $\gamma$, your algorithm might require too many steps to maximize $F$ if $\gamma$ is too small or it may not find the local maximum if $\gamma$ is too big.