I'm working with pressure-velocity coupled systems. It means that instead of solving 4 different linear systems in segregated approach (1 for pressure and 3 for Ux, Uy, Uz), we can solve only one coupled system. Assuming simple 3D rectangular mesh resulting matrix appears to be 7-diagonal with each element is a block 4x4 of arrow form \begin{pmatrix} a_{11} & 0 & 0 & a_{14} \\ 0 & a_{22} & 0 & a_{24} \\ 0 & 0 & a_{33} & a_{34} \\ a_{41} & a_{42} & a_{43} & a_{44} \end{pmatrix}

I would like to make efficient GPU-based BiCGStab solver for it, but it seems, that I have problem with preconditioning. On CPU BiCGStab + ILU or AMG works just fine. But on GPU, I can't use ILU family, because they are not suitable. I've tried AINV preconditioner, but it's either too slow and big or not robust enough. Diagonal preconditioner isn't effective at all. Now I'm thinking of polynomial, though I expect it to have similar problems as AINV.

Are there any other options that could possibly work with my approach, or the whole idea of GPU BiCGStab is a dead end in the first place?

  • $\begingroup$ How much faster is a single BiCGStab iteration on your GPU with say a diagonal preconditioner? Do you have a decent matvec implementation that makes it worth moving the sparse matrix over and iterating at all? $\endgroup$ – Bill Barth Aug 7 '15 at 17:12
  • $\begingroup$ Unpreconditioned GPU iteration about 20 times faster than unpreconditioned CPU. I had a thought about just using unprec version, but in this case we can easily have > 1000 iteration and even BiCGStab can be quite unstable on that long distance. Previously I worked with reformulated PCG + AINV and it was pretty good. $\endgroup$ – soh Aug 7 '15 at 17:44
  • $\begingroup$ "But on GPU, I can't use ILU family, because they are not suitable." Could you explain what you mean by "not suitable"? (Disclaimer: I know very little about GPUs.) Thanks! $\endgroup$ – nukeguy Jun 8 '16 at 19:16

Both ILU and diagonal scaling are not efficient preconditioners for "real" problems, i.e., if you let the size of your discrete problem become large. In other words, if you insist on timing GPUs vs CPUs with these preconditioners, you are only considering academic problems that are not really of interest -- if the problem is small enough that the use of these preconditioners make sense, then the problem is so small that nobody actually cares about the run time in practice.

If you do care about run time (because run time is an obstacle to doing actual simulations, not just because you want to do an academic timing study without practical implications), then you need to use preconditioners that make sense for large problems -- and these are typically based on the physical structure of the underlying problem, not just on the matrix values. An excellent example is the one by Silvester and Wathen. We implement and discuss its use in step-22 of the deal.II tutorial (see https://www.dealii.org/developer/doxygen/deal.II/step_22.html); pay particular attention to the discussion in the "Possibilities for extensions" section at the end of the "Results" section.

  • $\begingroup$ Thanks a lot for your answer, the link you gave was really interesting and I have a lot to think now. It seems I should move to AMG preconditioning. But I don't agree with you about your speech on real tasks. This work is a part of OpenFOAM framework, which is used worldwide for real industry cases. And I'm making GPU-accelerated version of already existing approach. For example I have 10 million (mostly hexahedra) cells test case for my work, which is solved on CPU with BiCGStab + Cholesky preconditioner. $\endgroup$ – soh Aug 13 '15 at 13:27
  • $\begingroup$ Then I would be very interested to seeing how your solver+preconditioner compares in terms of run time against the Silvester/Wather+AMG preconditioner with GMRES :-) $\endgroup$ – Wolfgang Bangerth Aug 13 '15 at 21:44
  • $\begingroup$ @WolfgangBangerth "Both ILU and diagonal scaling are not efficient preconditioners for "real" problems, i.e., if you let the size of your discrete problem become large." Do you have a reference that backs up this statement (namely the ILU part)? Thanks! $\endgroup$ – nukeguy Jun 8 '16 at 19:19
  • $\begingroup$ None at hand, but the literature has many examples. The only preconditioners that are truly scalable for PDEs are multigrid or variations thereof. That said, ILU is a rather effective smoother for multigrid iterations. $\endgroup$ – Wolfgang Bangerth Jun 8 '16 at 22:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.