# Best way of porting code from the GPU to MPI-nodes

I have a program, structured in two parts, $$A$$ and $$B$$. Both parts are capable of running as standalone units, and written in C++. $$A$$ is written for cluster systems, running entirely on CPU-nodes, connected via MPI (both on the same CPU for multiple cores and for different nodes), and $$B$$ is targeted to run on a single GPU. $$A$$ is doing FEM-calculations, while $$B$$ is only doing matrix-vector-multiplications, with a static dense matrix and varying vectors (matrix-size is typically 5kx5k-15kx15k complex double elements). The result of the matrix-vector-multiplication must be accessible from all threads in $$A$$.

In order to keep the memory load on the GPU as low as possible my strategy until now is to create the static matrix only in the first MPI thread in $$A$$, but the result matrix in all threads (giving me a large data chunk in GPU memory for the first thread, and small data chunks for all other threads) by creating as many instances of $$B$$ as there are threads in $$A$$, but only generating the matrix for one instance. After doing the multiplications in thread 0, the result is distributed using MPI to all the other threads. $$A$$ does calculations based on that result, and afterwards restarts $$B$$ with different parameters.

Now I have to port $$B$$ to the same nodes as $$A$$, without a GPU. I was looking at PBLAS-functions (PZGEMV) or the PLASMA-library, and at the Trilinos-library (which I already use in $$A$$, thus integration in $$B$$ should be easy), but there I am not sure if dense distributed matrix-vector multiplications are available. What would be the best strategy for porting, and to make the resulting program as efficient as possible?

• How large is this static matrix? Could it make sense to have each process store a copy of the static matrix and perform the matvecs locally so you minimize communication costs? If the matvecs are time consuming enough, could you bundle a few cores into a single MPI process and use OpenMP or some library to do parallel matvecs within that bundled process and then send out the results to the other processes in your communicator? – spektr Jul 12 at 22:25
• As written in the question, the static matrix has a size between 25kk and 225kk elements (resulting in sizes between 400 Mbyte and 4 GByte, roughly). Thus, for the smaller matrix it is possible to share it, for the larger matrix I would like to avoid sharing it for each thread. Concerning bundling: I am not sure if I can bundle cores for OpenMP-procedures when each of them is running a MPI-thread – arc_lupus Jul 13 at 6:41
• MPI+OpenMP is a common paradigm and storing one copy of the big matrix per node is desirable since communication costs are so much higher than computation. Try 1 MPI process per node with one OpenMP thread per core reach running a matvec. Alternatively, you could bundle your vectors into a smaller matrix and run a matmult for improved efficiency. – Richard Jul 14 at 13:43