I want to minimize an energy of the form $$V_1(\mathbf{x}) + V_2(\mathbf{x})$$ where $V_1$ is much stiffer than $V_2$. When I try to use LBFGS, convergence is extremely slow, as the solution oscillates wildly in the "stiff directions" while searching for the optimum in the non-stiff directions.

Is there a way I can modify how I am formulating the problem to avoid this problem? $V_1$ is nonlinear so the stiff directions change during optimization, though from the structure of the problem I can identify them easily at any given iteration.

  • $\begingroup$ How big is $x$? Do you really need to use a limited memory method or could you use full BFGS? Can you get an exact Hessian for $V_{1}$? Can you reparameterize the problem to separate the stiff directions from the not stiff directions? $\endgroup$ – Brian Borchers Dec 4 '16 at 17:42
  • $\begingroup$ @BrianBorchers $x$ contains ~100,000 variables. Unfortunately not (the code for computing $\nabla V_1$ is already quite slow) but I have a good sense of the directions in which $V_1$ increases slowly and quickly. $\endgroup$ – user168715 Dec 4 '16 at 18:10
  • $\begingroup$ Have you tried simply increasing the number of past gradients included in your L-BFGS computation? (10 is a common default, but you could try a larger number to see how that effects the results. The hope would be that you'd have enough curvature information to learn to avoid steps away from the "bottom of the valley.") Also, how precise a line search are you using? $\endgroup$ – Brian Borchers Dec 4 '16 at 23:52
  • $\begingroup$ I tried 20 but it did not noticeably change the results. What kind of tweaks do you suggest I try to the line search? By the way, the times when LBFGS seems to stall are near the optima of my function. This is surprising to me, since I thought that close to the optima, the (quasi-)Newton methods should work especially well. $\endgroup$ – user168715 Dec 5 '16 at 7:48
  • $\begingroup$ THe problem here is that your objective is very badly conditioned and L-BFGS isn't properly estimating the second derivatives/curvature. Full BFGS would probably do better, but you can't really afford that. $\endgroup$ – Brian Borchers Dec 5 '16 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.