17
Let me first answer why I think C++ interfaces to MPI have generally not been overly successful, having thought about the issue for a good long time when trying to decide whether we should just use the standard C bindings of MPI or building on something at higher level:
When you look at real-world MPI codes (say, PETSc, or in my case deal.II), one finds ...
13
I have always thought that we should use it in our own project, deal.II, because it is higher level than pure MPI and can save a few lines of code here and there. That said, what I learned over the years is that most high-level code doesn't actually have that much MPI code to begin with -- the 600,000 lines of code in deal.II have only ~50 calls to MPI. That'...
12
I think the answer to this is no. Once you've pushed them into the MPI stack, they're out of your control, and the MPI semantics govern the way the messages are sent.
You could certainly prioritize messages by queuing them in your code before sending them, and then checking frequently which are the most important to send. But I'm not at all convinced that ...
12
Others have already proposed the various MPI_Probe variants but I'd like to point out one thing: MPI is not a remote procedure call, i.e., there are no ways to notify a process that some message has come in (e.g., by raising a signal). Messages are sent but if the receiving process doesn't actually go look for them, then nothing will happen. As such, the ...
12
Good is a relative term, and it will depend on the nature of the problem, the nature of the algorithm, and properties of the hardware involved. The only absolute reference point is ideal scaling (100% efficiency).
You can claim your scaling is good if it is better than what anyone else has achieved for the same problem, or if it's "close" to ideal for ...
11
Remarkably, MPI_Ibarrier is a very useful routine. For example, you can deliver an unstructured round of messages to ranks that do not know how many messages to receive by sending with MPI_Issend (yes, a rare use of synchronous send), then entering a loop of alternating MPI_Testall (to see if the sends completed) and MPI_Iprobe (to process incoming messages)....
11
The conjugate gradient method is the provably fastest iterative solver, but only for symmetric, positive-definite systems. What would be awfully convenient is if there was an iterative method with similar properties for indefinite or non-symmetric matrices.
The CG method seeks approximate solutions at each step $k$ within the Krylov subspace
$K_k(A,b) = \{...
10
The Thomas algorithm is very efficient because its operation count is very low and because data accesses are very likely to be cache hits once data is initially read from memory.
There are two loops.
The first loop traverses the data forward. Each element of the lower, main and upper triangle, along with the right-hand-side vector (which is typically ...
10
A long standing favorite benchmark in high performance computing has been the HPLinpack benchmark, which measures the speed of a computer system in floating point operations per second while solving a very large, dense, linear system of equations. It is assumed that the solution takes $2/3n^{3}+2n^{2}$ floating point operations and the tester is allowed to ...
10
If the operation is as trivial as you say, and each node has all the information necessary to carry out the operation, then the communication will be substantially more expensive than recomputing locally on each node. That said, it's a good exercise to write both implementations and compare; experiment is better than the advice of strangers on the internet.
...
9
First, the exact answer depends on: (1) usage, i.e. function input arguments, (2) MPI implementation quality and details, and (3) the hardware you're using. Often, (2) and (3) are related, such as when the hardware vendor optimizes MPI for their network.
In general, fusing MPI collectives is better for smaller messages, since start-up costs can be ...
8
See http://www.unixer.de/research/nbcoll/, especially http://www.unixer.de/research/nbcoll/libnbc/, which is the reference implementation that sits on top of MPI-1. As Jed said, MPICH2 1.5 already has NBC and the implementation is similar to Torsten's (although almost certainly faster due to lower overhead).
8
I'm a happy user of GoogleTest with a C++ MPI code in a CMake/CTest build environment:
CMake automatically installs/links googletest from svn!
adding tests is a one-liner!
writing the tests is easy! (and google mock is very powerful!)
CTest can pass command-line parameters to your tests, and exports data to CDash!
This is how it works. A batch of unit-...
8
I think some of your issues are more important than others and some of
your emphasis is misplaced. In pursuing overhead, you are in danger of
making your program unmaintainable. It is easier to write a common
program and direct surplus effort somewhere more interesting. I apologize for pontificating like this.
If statements. From a strict programming ...
7
Currently MPI has no provisions for prioritisation of messages and neither has the upcoming MPI 3.0 standard. It is up to the MPI implementation to decide how to transmit the messages. E.g. smaller messages might get sent faster because of certain bypasses in the communication machinery (highly implementation and system dependent). You might be able to ...
7
In alphabetical order (disclaimer: I am the main author of Elemental):
DPLASMA
Distributed Parallel Linear Algebra Software for Multicore Architectures (DPLASMA) is a relatively recent and ongoing effort by Bosilca et al. to extend PLASMA to distributed-memory machines. Version 1.0.0 supports distributed Cholesky factorizations, among many other operations....
7
You could try having all processors use MPI_IProbe or MPI_Probe with MPI_ANY_SOURCE to check if there are any receivable messages with a given tag. If there are matching messages, you can extract the senders rank from the returned status and call MPI_Recv immediately.
7
http://mpitutorial.com/tutorials/dynamic-receiving-with-mpi-probe-and-mpi-status/ has a tutorial describing the use of MPI_Probe that might be useful to you.
6
As far as I know, boost::mpi is just a c++ wrapper around the C API. As such, you should be able to use boost::mpi and switch to the C API whenever some functionality is not implemented. Indeed, from their webpage:
The thin abstractions in Boost.MPI allow one to easily combine it with calls to the underlying C MPI library.
I have not used it myself, and ...
6
You seem to have the wrong declaration for stat. It must be declared as an array of size MPI_STATUS_SIZE.
integer stat(MPI_STATUS_SIZE)
6
To get the ball rolling, here are two of my needs:
The interface should be able to eliminate redundant or unnecessary arguments, e.g. MPI_IN_PLACE.
The interface should auto-detect built-in datatypes ala Elemental's MpiMap.
If/whenever possible, user-defined datatypes should be constructed for classes.
6
There's no guarantee in the standard that any progress is made on the non-blocking sends until you actually call MPI_WAIT. It's a perfectly valid implementation to just queue up the operations and when you call MPI_WAIT, all of the MPI_ISEND operations complete at once. In reality, they usually tend to get a chance to progress anytime you enter the MPI ...
6
Bill answered the first part, so I'll only answer the second question. An MPI send is blocking if it does not return until it is safe to modify the send buffer and a receive is blocking if it does not return until the receive buffer contains the newly-received message. In practice, outside of buffered sends (thanks, Hristo Iliev), this implies that ...
6
Always use the correct types as specified by the standard. MPI_Comm for your communicators not int unless you're in Fortran. Etc., etc.
This should be relatively easy. What problems are you really having?
Edited to add in response to update: It looks like the type of childGroup is not an MPI_Group but an MPI_Comm, so you should either fix that or use a ...
6
Bill Barth already gave great advice. My advice is to not read the MPICH or OpenMPI documents. Read the MPI standard instead. As far as standards are concerned, it's actually quite readable. Implementations are, in my experience, quite conforming to the standard, but they add and extend them. These extensions are part of the problem that makes code ...
6
The honest answer is that we don't know. The answer depends heavily on what is actually being run and what code the user has written. As Brian Borchers points out, there's a big difference between two benchmarks where we have all the code and supposedly know what that code is doing, but there's much disagreement about how representative this code is of what ...
6
The first thing you need to ask yourself: is your problem big enough that the overhead of MPI messaging is less than the work that you save. Your problem size is 10k which is small, but on the other hand you seem to have a dense matrix, so there is a good amount of work.
Next: parallelizing a numerical method may change the mathematics. It looks like you're ...
5
There are a number of possible solutions to this, I would recommend that you commit a new MPI data type using MPI_Type_struct(). If you have multiple matrices of different sizes, you'll need to commit a new data type for each (the data type size is static) or consider a more flexible approach.
Only top voted, non community-wiki answers of a minimum length are eligible
