Assuming that the underlying algorithm can be ported to multi GPU, what aspects should one consider while porting from MPI (on multiple nodes) to multi GPU (again on multiple nodes)?
Making use of Hyper Q ? Of MPS ? Peer to peer communication ?
For example, if we consider an MPI code that solves for the Navier Stokes equations using ADI, on multiple nodes with domain decomposition, and solves for the pressure and velocity by solving the schur complement matrices and interface unknowns induced by the domain decomposition. We could solve for internal unknowns by a simple tridiagonal solver (say, Thomas algorithm) which is highly parallel and use the schur complement method to solve for the interface unknowns and repeat this throughout the time stepping.
The schur complement would involve solving directional systems (MPI ranks in depending on the cartesian decomposition in one direction) and hence would involve communication between the ranks in one direction.
The issue of communication arises when porting to the GPU, as we would need to communicate between the GPU's on different nodes which are logically in the same direction (considering 1:1 ratio of MPI ranks: GPU).
Any suggestions or ideas are welcome.