2
$\begingroup$

I have a need to get a variable value from another process rank of which I know. This happens in context of a parallel solver of a "A x = b" equation, for which process with rank 0 knows matrix A, and the other processes put some values (i,j) from this matrix into the matrix of a parallel solver data type (Petsc's Mat). This means that these other processes walk through the range delegated to them, calculate i and j, retrieve ij's element, and call MatSetValue. There is no way to avoid such way - it is a finite differencing method and the parent process has variable values in central points and in neighbouring points (left,right,top,bottom) For example the same happens in ex13F90.F in petsc library examples.

The problem is that I don't know a proper MPI subroutine to retrieve the a_central, a_top, a_bottom, ..., values from the parent processes. Right now I tried to broadcast them (MPI_Bcast) but this means that each process has the entire matrix and runs out of memory.

Per the answer below, the essence of this question is "How do I do one-sized MPI communication?"...

Improve this question
$\endgroup$
1
  • $\begingroup$ Any time you say "There is no way to avoid such way", you're likely to be wrong. In the case of "a finite differencing method", I know you are wrong because people have been doing finite-differencing and in PETSc for 20 years without ever touching one-sided communication. $\endgroup$
    – Jeff Hammond
    Oct 20, 2013 at 21:01

2 Answers 2

Reset to default
3
$\begingroup$

There are generally three approaches to this problem:

1/ If processor P has a deterministic algorithm by which it needs to know which matrix elements it needs, then processor 1 can replay this same algorithm to find out which elements processor P will need. It can then send these elements to processor P (and P alone, not everyone).

2/ Processor P can "simulate" the algorithm by which it does computations and record which matrix elements it will need to do its work. It can then send this list of elements it will need to processor 1 and processor 1 can reply by sending these elements to P. P can then perform its algorithm.

3/ Processor P can use "one-sided" MPI communication to simply fetch data elements from the memory of processor 1.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Approach 3/ - how? $\endgroup$
    – user5273
    Oct 17, 2013 at 0:58
  • $\begingroup$ @Svetlana You should look at MPI_Put and MPI_Get. These allow for a process to access another process memory space without requiring both processes participate in the communication operation. I would point you to the one-sided communication section of the MPI-3 standard for more information: mpi-forum.org/docs/mpi-3.0/mpi30-report.pdf As the MPI-3 standard is still fairly new, vendors may not have all of it implemented. You may need to use a different implementation; the latest version of MPICH has one-sided communication. $\endgroup$
    – Brian Skjerven
    Oct 17, 2013 at 15:40
  • $\begingroup$ @BrianSkjerven: this means using MPI windows, is there no way to just retrieve a variable value from another process without them? Seems complicated. (I would keep trying, but it does seem like someone was overthinking something.) $\endgroup$
    – user5273
    Oct 18, 2013 at 0:33
  • $\begingroup$ @Svetlana For one-sided communication, no, you have to use windows. Wolfgangs's first two approaches could be accomplished without one-sided operations. If I understand your method correctly, couldn't you use a combination of MPI_Gather/Scatter (or the vector versions)? $\endgroup$
    – Brian Skjerven
    Oct 18, 2013 at 3:33
  • $\begingroup$ @Brian That's right. I'm looking for examples of that: seems like a common thing to do in a solver. $\endgroup$
    – user5273
    Oct 28, 2013 at 0:27
0
$\begingroup$
  • Step 1: Hack inside of the PETSc matrix object and add an MPI_WIN_CREATE call when the data is allocated.
  • Step 2: Write an address translation routine that converts from matrix row and column coordinates (i,j) to a process and a remote offset to the window (proc,offset). A ranged version of this function is highly desirable.
  • Step 3: Use the function from Step 2 to do MPI_{PUT,GET} against your PETSc matrix. This is probably best done with new API functions for said matrix object.

For the case of dense matrices, you are basically implementing Global Arrays 3.0 :-)

If you want an answer to your ultimate question ("How do I do one-sided MPI communication"), please make a separate post about that so it is clear to others what the question really is.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.