30
There is one real and one practical reason.
First, MPI was developed at a time when machines had exactly one processor core and when we wanted to couple different machines. It is today used on clusters of tens of thousands of machines, each of which happens to have many cores but the point is that it's still separate machines. Now, a processor core on ...
18
Let me first answer why I think C++ interfaces to MPI have generally not been overly successful, having thought about the issue for a good long time when trying to decide whether we should just use the standard C bindings of MPI or building on something at higher level:
When you look at real-world MPI codes (say, PETSc, or in my case deal.II), one finds ...
13
I have always thought that we should use it in our own project, deal.II, because it is higher level than pure MPI and can save a few lines of code here and there. That said, what I learned over the years is that most high-level code doesn't actually have that much MPI code to begin with -- the 600,000 lines of code in deal.II have only ~50 calls to MPI. That'...
12
Others have already proposed the various MPI_Probe variants but I'd like to point out one thing: MPI is not a remote procedure call, i.e., there are no ways to notify a process that some message has come in (e.g., by raising a signal). Messages are sent but if the receiving process doesn't actually go look for them, then nothing will happen. As such, the ...
12
Remarkably, MPI_Ibarrier is a very useful routine. For example, you can deliver an unstructured round of messages to ranks that do not know how many messages to receive by sending with MPI_Issend (yes, a rare use of synchronous send), then entering a loop of alternating MPI_Testall (to see if the sends completed) and MPI_Iprobe (to process incoming messages)....
12
The conjugate gradient method is the provably fastest iterative solver, but only for symmetric, positive-definite systems. What would be awfully convenient is if there was an iterative method with similar properties for indefinite or non-symmetric matrices.
The CG method seeks approximate solutions at each step $k$ within the Krylov subspace
$K_k(A,b) = \{...
12
Good is a relative term, and it will depend on the nature of the problem, the nature of the algorithm, and properties of the hardware involved. The only absolute reference point is ideal scaling (100% efficiency).
You can claim your scaling is good if it is better than what anyone else has achieved for the same problem, or if it's "close" to ideal for ...
10
The Thomas algorithm is very efficient because its operation count is very low and because data accesses are very likely to be cache hits once data is initially read from memory.
There are two loops.
The first loop traverses the data forward. Each element of the lower, main and upper triangle, along with the right-hand-side vector (which is typically ...
10
A long standing favorite benchmark in high performance computing has been the HPLinpack benchmark, which measures the speed of a computer system in floating point operations per second while solving a very large, dense, linear system of equations. It is assumed that the solution takes $2/3n^{3}+2n^{2}$ floating point operations and the tester is allowed to ...
10
If the operation is as trivial as you say, and each node has all the information necessary to carry out the operation, then the communication will be substantially more expensive than recomputing locally on each node. That said, it's a good exercise to write both implementations and compare; experiment is better than the advice of strangers on the internet.
...
9
I'm a happy user of GoogleTest with a C++ MPI code in a CMake/CTest build environment:
CMake automatically installs/links googletest from svn!
adding tests is a one-liner!
writing the tests is easy! (and google mock is very powerful!)
CTest can pass command-line parameters to your tests, and exports data to CDash!
This is how it works. A batch of unit-...
9
Wolfgang Bangerth's answer is totally correct, and I only want to add one practical aspect.
Portability across hardware
Let's say you write a research code from scratch. You have a powerful multi-core shared memory machine at your department which can do the job. If you start out with a thread-based implementation, you can reach good performance on this ...
8
See http://www.unixer.de/research/nbcoll/, especially http://www.unixer.de/research/nbcoll/libnbc/, which is the reference implementation that sits on top of MPI-1. As Jed said, MPICH2 1.5 already has NBC and the implementation is similar to Torsten's (although almost certainly faster due to lower overhead).
8
I think some of your issues are more important than others and some of
your emphasis is misplaced. In pursuing overhead, you are in danger of
making your program unmaintainable. It is easier to write a common
program and direct surplus effort somewhere more interesting. I apologize for pontificating like this.
If statements. From a strict programming ...
7
http://mpitutorial.com/tutorials/dynamic-receiving-with-mpi-probe-and-mpi-status/ has a tutorial describing the use of MPI_Probe that might be useful to you.
7
You could try having all processors use MPI_IProbe or MPI_Probe with MPI_ANY_SOURCE to check if there are any receivable messages with a given tag. If there are matching messages, you can extract the senders rank from the returned status and call MPI_Recv immediately.
7
In alphabetical order (disclaimer: I am the main author of Elemental):
DPLASMA
Distributed Parallel Linear Algebra Software for Multicore Architectures (DPLASMA) is a relatively recent and ongoing effort by Bosilca et al. to extend PLASMA to distributed-memory machines. Version 1.0.0 supports distributed Cholesky factorizations, among many other operations....
6
You seem to have the wrong declaration for stat. It must be declared as an array of size MPI_STATUS_SIZE.
integer stat(MPI_STATUS_SIZE)
6
As far as I know, boost::mpi is just a c++ wrapper around the C API. As such, you should be able to use boost::mpi and switch to the C API whenever some functionality is not implemented. Indeed, from their webpage:
The thin abstractions in Boost.MPI allow one to easily combine it with calls to the underlying C MPI library.
I have not used it myself, and ...
6
To get the ball rolling, here are two of my needs:
The interface should be able to eliminate redundant or unnecessary arguments, e.g. MPI_IN_PLACE.
The interface should auto-detect built-in datatypes ala Elemental's MpiMap.
If/whenever possible, user-defined datatypes should be constructed for classes.
6
There's no guarantee in the standard that any progress is made on the non-blocking sends until you actually call MPI_WAIT. It's a perfectly valid implementation to just queue up the operations and when you call MPI_WAIT, all of the MPI_ISEND operations complete at once. In reality, they usually tend to get a chance to progress anytime you enter the MPI ...
6
Bill answered the first part, so I'll only answer the second question. An MPI send is blocking if it does not return until it is safe to modify the send buffer and a receive is blocking if it does not return until the receive buffer contains the newly-received message. In practice, outside of buffered sends (thanks, Hristo Iliev), this implies that ...
6
Always use the correct types as specified by the standard. MPI_Comm for your communicators not int unless you're in Fortran. Etc., etc.
This should be relatively easy. What problems are you really having?
Edited to add in response to update: It looks like the type of childGroup is not an MPI_Group but an MPI_Comm, so you should either fix that or use a ...
6
Bill Barth already gave great advice. My advice is to not read the MPICH or OpenMPI documents. Read the MPI standard instead. As far as standards are concerned, it's actually quite readable. Implementations are, in my experience, quite conforming to the standard, but they add and extend them. These extensions are part of the problem that makes code ...
6
The honest answer is that we don't know. The answer depends heavily on what is actually being run and what code the user has written. As Brian Borchers points out, there's a big difference between two benchmarks where we have all the code and supposedly know what that code is doing, but there's much disagreement about how representative this code is of what ...
6
The first thing you need to ask yourself: is your problem big enough that the overhead of MPI messaging is less than the work that you save. Your problem size is 10k which is small, but on the other hand you seem to have a dense matrix, so there is a good amount of work.
Next: parallelizing a numerical method may change the mathematics. It looks like you're ...
5
There's nothing surprising about these results. Matrix multiplication is well known to be communication intensive, and you've got a relatively slow communications network between your four nodes.
Using MPI between two processes on the same node is certainly faster than using MPI between processes on different nodes because you don't have the bandwidth ...
5
Overlapping windows are permitted in MPI-2. In fact, the RMA working group actually believes that this is a sufficiently reasonable usage that they refused better mechanisms for switching between usage modes (via window attributes).
It is unrelated, but I'm the person who got request-based RMA into MPI-3. And I had to fight like hell for them.
5
This is not supported by MPI nor any other communication middleware I know of. This is probably because it's not supported by any hardware I know of, with the exception of Blue Gene, where there are high-priority packets for control messages that will overtake other messages under some conditions. However, these are not for general use since they only ...
5
Sometimes you can get an alternative view on performance problems via a high-level resource analysis: Is there a relevant bottleneck such as memory bandwidth? Does every worker thread do the same amount of work? This data might be collected easily with likwid-perfctr from the LIKWID tool suite LIKWID Google code project. If the profile is such that many ...
Only top voted, non community-wiki answers of a minimum length are eligible
