When using many iterative methods, whether for solving linear systems, looking for steady-state convergence in CFD, etc., the semilog plot of the residual often shows "humps" as the residual decays. These humps generally appear in a regular pattern (same number of iterations between them for example). I've seen cases where the valley between the hump can be several orders of magnitude lower than the peak of the hump.

For example, consider this image of a residual vs. iteration count (just the black line is important):

enter image description here

Another example that I can't embed here is Figure 4 in this AIAA paper.

What is it that causes these humps in the residual? Why do they appear with a somewhat regular frequency?

  • $\begingroup$ This strongly depends on the method. What is the iterative method you are using here? I'm saying this because you don't typically see this behavior with the following iterative methods: CG, fixed point iteration, Newton iteration. $\endgroup$ Oct 1 '15 at 2:23
  • $\begingroup$ @WolfgangBangerth In this case, it's using a dual-time scheme where the inner iteration is a commonly used explicit scheme for a viscous fluid problem. But I recall seeing this behavior with other schemes (ADI and sometimes AUSM) in fluids, and I recall seeing it with linear algebra solvers and eigenvalue solvers I had written before. I'm sure some methods show it and some don't, but I've seen it in several different domains. I wish I could find more figures online that show it. $\endgroup$
    – tpg2114
    Oct 1 '15 at 2:37

These humps occur in the context of optimization when the solution is approached but we then go past it.

This frequently occurs in gradient descent with momentum, where they appear regularly on plots because the gradient and momentum both decay in magnitude together. Usually this suggests a little too much momentum, since we're retaining speed as we reach the solution. If the amount of momentum retained each iteration is reduced enough, you can get the humps to disappear and for iterations to smoothly descend much faster.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.