Assuming that the underlying algorithm can be ported to multi GPU, what aspects should one consider while porting from MPI (on multiple nodes) to multi GPU (again on multiple nodes)?

Making use of Hyper Q ? Of MPS ? Peer to peer communication ?

For example, if we consider an MPI code that solves for the Navier Stokes equations using ADI, on multiple nodes with domain decomposition, and solves for the pressure and velocity by solving the schur complement matrices and interface unknowns induced by the domain decomposition. We could solve for internal unknowns by a simple tridiagonal solver (say, Thomas algorithm) which is highly parallel and use the schur complement method to solve for the interface unknowns and repeat this throughout the time stepping.

The schur complement would involve solving directional systems (MPI ranks in depending on the cartesian decomposition in one direction) and hence would involve communication between the ranks in one direction.

The issue of communication arises when porting to the GPU, as we would need to communicate between the GPU's on different nodes which are logically in the same direction (considering 1:1 ratio of MPI ranks: GPU).

Any suggestions or ideas are welcome.

Thank You.

  • 3
    $\begingroup$ I find this question impossible to answer. MPI and GPUs use entirely different paradigms and strategies to parallelize. There is no easy answer for "what to look out for" -- you will have to fundamentally redesign your algorithms to port from MPI to GPU. There is no one single issue anyone can list in answering your question. $\endgroup$ Oct 31 '16 at 19:37
  • $\begingroup$ I am sorry but I do not quite understand when you say that a single issue cannot be listed. I understand that MPI and GPU's use different paradigms. But as I see it, a program with high amount of parallelism could be beneficial to run on a GPU. In a supercomputing cluster with a couple GPU's per node, the GPU's are used as accelerators. The example of the Navier Stokes equation above where each rank calculates its own field (u,v,w,p,rho) with some communication, can it not be reprogrammed to the GPU without much effort ? Because the basic idea is the solution of tridiagonal systems, $\endgroup$
    – Mathnoob
    Nov 1 '16 at 10:11
  • $\begingroup$ which are have a lot of parallelism. $\endgroup$
    – Mathnoob
    Nov 1 '16 at 10:12
  • $\begingroup$ Yes, programs with a lot of parallelism are well suited for GPUs. The problem is that for typical flow simulations, you trade the amount of parallelism for the amount of communication: The more cores you want to use, the more you need to communicate. Experience in the community is that it's very difficult to create enough parallelism for the hundreds or thousands of cores on GPUs without losing all efficiency because of the amount of communication that involves. $\endgroup$ Nov 2 '16 at 19:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.