2
$\begingroup$

I have a program, structured in two parts, $A$ and $B$. Both parts are capable of running as standalone units, and written in C++. $A$ is written for cluster systems, running entirely on CPU-nodes, connected via MPI (both on the same CPU for multiple cores and for different nodes), and $B$ is targeted to run on a single GPU. $A$ is doing FEM-calculations, while $B$ is only doing matrix-vector-multiplications, with a static dense matrix and varying vectors (matrix-size is typically 5kx5k-15kx15k complex double elements). The result of the matrix-vector-multiplication must be accessible from all threads in $A$.

In order to keep the memory load on the GPU as low as possible my strategy until now is to create the static matrix only in the first MPI thread in $A$, but the result matrix in all threads (giving me a large data chunk in GPU memory for the first thread, and small data chunks for all other threads) by creating as many instances of $B$ as there are threads in $A$, but only generating the matrix for one instance. After doing the multiplications in thread 0, the result is distributed using MPI to all the other threads. $A$ does calculations based on that result, and afterwards restarts $B$ with different parameters.

Now I have to port $B$ to the same nodes as $A$, without a GPU. I was looking at PBLAS-functions (PZGEMV) or the PLASMA-library, and at the Trilinos-library (which I already use in $A$, thus integration in $B$ should be easy), but there I am not sure if dense distributed matrix-vector multiplications are available. What would be the best strategy for porting, and to make the resulting program as efficient as possible?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ How large is this static matrix? Could it make sense to have each process store a copy of the static matrix and perform the matvecs locally so you minimize communication costs? If the matvecs are time consuming enough, could you bundle a few cores into a single MPI process and use OpenMP or some library to do parallel matvecs within that bundled process and then send out the results to the other processes in your communicator? $\endgroup$ – spektr Jul 12 '19 at 22:25
  • $\begingroup$ As written in the question, the static matrix has a size between 25kk and 225kk elements (resulting in sizes between 400 Mbyte and 4 GByte, roughly). Thus, for the smaller matrix it is possible to share it, for the larger matrix I would like to avoid sharing it for each thread. Concerning bundling: I am not sure if I can bundle cores for OpenMP-procedures when each of them is running a MPI-thread $\endgroup$ – arc_lupus Jul 13 '19 at 6:41
  • $\begingroup$ MPI+OpenMP is a common paradigm and storing one copy of the big matrix per node is desirable since communication costs are so much higher than computation. Try 1 MPI process per node with one OpenMP thread per core reach running a matvec. Alternatively, you could bundle your vectors into a smaller matrix and run a matmult for improved efficiency. $\endgroup$ – Richard Jul 14 '19 at 13:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.