2
$\begingroup$

I am solving a system of nonlinear time-dependent equations using the Newton method in a finite-element-setting, i.e. first I create the jacobian matrix for the current time, and afterwards I try to solve the system (using a GMRES-solver, preconditioned using the AMG-method) to get the solution update. This is repeated until the residual does not get smaller anymore, then the time is increased by one time step. For each time step the necessary GMRES iterations are recorded.
Now I noticed that initially the GMRES iterations stay constant at 50 iterations for each time step, but after a certain time (depending on the input parameters) they increase suddenly to 75-100 iterations, and in the next time step the solver fails to converge at all (even after >10000 iterations).
Several suggestions are given here: Why is my iterative linear solver not converging?, but implementing those suggestions will take some time, thus, if there are already some hints based on the behaviour I could use giving me an initial idea, that would reduce the time necessary for finding the bug.
Furthermore, using a direct solver is not possible, after I currently am using > 40kk elements (which is way to big for the memory available). In order to calculate the equations correctly without introducing oscillations, I can not reduce the amount of elements.

Concerning the comments:

  • I am discretizing my time step using the Crank-Nicholson-method, to avoid instabilities
  • My system consists out of three coupled non-linear heat-equation-like equations, which are strongly coupled during solving
  • I am using a fixed time step and a fixed mesh density
  • My system has ~40e6 degrees of freedom. Usually (for smaller systems) I allow the solver to use as many iterations as I have degrees of freedom, but here I wanted to investigate the behavior of the system, thus I reduced the amount of allowed iterations. In addition, ~50 iterations take ~50 seconds, thus allowing 40e6 iterations will stop solving the program entirely.
  • Currently I am using 28 krylov vectors (default setting of the solver), but can change that. If there are suggestions for the size I should use instead, I can implement that, though
  • Preconditioning is from the left
Improve this question
$\endgroup$
7
  • $\begingroup$ If I'm understanding you correctly, you're using Newton's method for the nonlinear implicit time step? What's you're time discretization. Can you be more specific with your problem? What is you solution strategy? Do you have CFL ramping, what are you simulating... $\endgroup$
    – EMP
    Sep 1, 2019 at 9:00
  • $\begingroup$ Also, how many unknowns do you have? You say you did 10000 GMRES iterations, but how many krylov vectors per iteration? $\endgroup$
    – EMP
    Sep 1, 2019 at 12:15
  • $\begingroup$ @EMP: I hope I could address all your comments in the edited question $\endgroup$
    – arc_lupus
    Sep 2, 2019 at 5:47
  • $\begingroup$ Thanks for the update. But I think the most important piece of information is how many krylov vectors are you using? Restarted GMRES can stagnate, so if be curious to know how many krylov vectors per iteration and are you left or right or flexible preconditioned. $\endgroup$
    – EMP
    Sep 2, 2019 at 10:37
  • $\begingroup$ @EMP: Added, but those values can be changed. Unfortunately, I am not that familiar with those settings, thus suggestions for improvement are welcome. $\endgroup$
    – arc_lupus
    Sep 2, 2019 at 11:03

1 Answer 1

Reset to default
1
$\begingroup$

It looks to me like the most helpful thing you can do is increase the number of krylov vectors you are using. This is somewhat different from the typical behavior I've seen where the first step takes the longest, but maybe the physics are different enough. I think it's important here to understand how GMRES works, as some of your explanations revealed some misunderstandings. Saad wrote a great book with the algorithms linked here: https://www-users.cs.umn.edu/~saad/IterMethBook_2ndEd.pdf

You begin with a linear system: $$Ax = b$$ and you want to find an x vector such that $$r = b - Ax = 0$$ You begin with some guess for x (usually x = 0) and you get a residual vector $r_0 = b$ and this will be your search direction (v vector), so you normalize it and multiply it by the A matrix to get a w vector. This vector gets dotted with each previous search direction (v vector) and then the w vector gets subtracted from it the components that were along each previously used search direction. this w vector gets normalized and becomes the new search direction. you go up to the top and repeat. The thing to notice is that this is the description for the full GMRES solver and it is guaranteed to obtain the solution to the linear system after using at most the number of unknowns of the system for the search direction. It is obviously untenable to use this many search directions, as each search direction has n elements for the nxn system. This would mean that for n search directions we're storing a full nxn system, which is clearly expensive; so we use restarted GMRES. Restarted GMRES uses a given number of krylov vectors, usually far fewer than the number of columns, and will loop over this partial GMRES solver, now with $x_0 \neq 0$ and resuming with the last guess for $x$. Restarted GMRES is far more efficient memorywise, but it can stagnate, so oftentimes peoples will use various preconditioners or varying numbers of krylov vectors to help solve the stiffer problems.

Some additional things to know, GMRES will stop your iterative solver from diverging assuming your preconditioner does not diverge, but you want the majority of the work to be done by a strong preconditioner, with GMRES stopping your code from blowing up. There are a couple different ways to precondition GMRES; typically it is best to use flexible right preconditioning as it allows you to truncate mid iteration with the actual linear residual computed at minimal cost. Flexible preconditioning does require 2x as much memory as typical left or right preconditioned GMRES, but it is cheaper. So I'd suggest you increase the number of Krylov vectors and use flexible right preconditioning (usually referred to just as flexible), and the problem should go away. Hope this helps!

Improve this answer
$\endgroup$
6
  • $\begingroup$ I tested your suggestions (increase of the number of krylov vectors, and flexible/right preconditioning). When changing the preconditioning, the solver failed to converge immediately. When increasing the number of vectors, the solver convergence did not change at all. $\endgroup$
    – arc_lupus
    Sep 3, 2019 at 16:21
  • $\begingroup$ How many vectors did you use? $\endgroup$
    – EMP
    Sep 3, 2019 at 16:45
  • $\begingroup$ 28/40. Are there suggestions about the maximum limit of vectors I should use? I tried to find some in literature, but was not able to get any useful results yet. $\endgroup$
    – arc_lupus
    Sep 4, 2019 at 5:48
  • $\begingroup$ Well the maximum number of vectors is the number of DOFs in the nonlinear problem. This is as I said an insanely high value. When you say no change can you post convergence plots? How many GMRES iterations did you run with 40 vectors. Can you strengthen your preconditioner at all (maybe run more smoothing in the AMG)? $\endgroup$
    – EMP
    Sep 4, 2019 at 7:15
  • $\begingroup$ Should I just add the convergence plots in the OP? I did not change the maximum number of iterations when changing the amount of vectors, and the resulting GMRES-Iterations did not change, either. $\endgroup$
    – arc_lupus
    Sep 4, 2019 at 15:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.