32
votes
Accepted
Why would you need frameworks like MPI when you can multi-task using threads?
There is one real and one practical reason.
First, MPI was developed at a time when machines had exactly one processor core and when we wanted to couple different machines. It is today used on ...
- 50.2k
14
votes
Accepted
What's the difference between conjugate gradient method and biconjugate gradient method
The conjugate gradient method is the provably fastest iterative solver, but only for symmetric, positive-definite systems. What would be awfully convenient is if there was an iterative method with ...
- 8,087
12
votes
Accepted
What is "good" parallel scaling?
Good is a relative term, and it will depend on the nature of the problem, the nature of the algorithm, and properties of the hardware involved. The only absolute reference point is ideal scaling (100%...
- 16.2k
11
votes
Parallel vs Serial Thomas Algorithm
The Thomas algorithm is very efficient because its operation count is very low and because data accesses are very likely to be cache hits once data is initially read from memory.
There are two ...
- 131
10
votes
Communication overhead in supercomputing
A long standing favorite benchmark in high performance computing has been the HPLinpack benchmark, which measures the speed of a computer system in floating point operations per second while solving a ...
- 17.6k
10
votes
Accepted
Writing a parallel version of an algorithm. Only which someparts are worth distributing
If the operation is as trivial as you say, and each node has all the information necessary to carry out the operation, then the communication will be substantially more expensive than recomputing ...
- 8,087
9
votes
Why would you need frameworks like MPI when you can multi-task using threads?
Wolfgang Bangerth's answer is totally correct, and I only want to add one practical aspect.
Portability across hardware
Let's say you write a research code from scratch. You have a powerful multi-...
- 1,936
8
votes
Accepted
C - OpenMP, MPI, Serial Program
I think some of your issues are more important than others and some of
your emphasis is misplaced. In pursuing overhead, you are in danger of
making your program unmaintainable. It is easier to write ...
- 11.4k
6
votes
What's the difference between conjugate gradient method and biconjugate gradient method
The conjugate gradient method only works to solve the system
$ A x = b $
if $A$ is symmetric and positive-definite (also works for negative definite). The reason it must be symmetric is that ...
- 4,567
6
votes
Accepted
How does one write MPI-implementation-independent code?
Always use the correct types as specified by the standard. MPI_Comm for your communicators not int unless you're in Fortran. Etc., etc.
This should be relatively easy. What problems are you really ...
- 10.8k
6
votes
How does one write MPI-implementation-independent code?
Bill Barth already gave great advice. My advice is to not read the MPICH or OpenMPI documents. Read the MPI standard instead. As far as standards are concerned, it's actually quite readable. ...
- 50.2k
6
votes
Communication overhead in supercomputing
The honest answer is that we don't know. The answer depends heavily on what is actually being run and what code the user has written. As Brian Borchers points out, there's a big difference between two ...
- 10.8k
6
votes
MPI+OpenMP Scalability
No. You need to test it to that scale, especially if you have MPI calls within OpenMP regions.
- 10.8k
6
votes
Why my parallel code using MPI is much slower than the serial one?
The first thing you need to ask yourself: is your problem big enough that the overhead of MPI messaging is less than the work that you save. Your problem size is 10k which is small, but on the other ...
- 915
5
votes
What features do users need from an MPI C++ interface?
My list in no particular order of preference. The interface should:
be header only, without any dependencies but <mpi.h>, and the standard library,
be ...
- 1,075
5
votes
Distributed (MPI) matrix matrix multiplication
You say that you want an MPI version. Then you need to study the literature, as the distributed memory variant of matrix-matrix product are not a simple parallellization of the sequential version.
...
- 915
5
votes
Writing a parallel version of an algorithm. Only which someparts are worth distributing
An addition to @Daniel Shapero's answer.
It might also be important to know if there are computations that can lead to different results depending on which machine they are launched (or just vary ...
- 8,287
5
votes
CPU usage when a MPI rank waits during a blocking communication
It depends on the communication settings you use for MPI. In blocking communication, MPI has three wait modes.
Aggressive busy wait. This is a kind of default mode. Open MPI, at least, uses this when ...
- 3,091
4
votes
Accepted
Parallel 2d heat equation (implicit timestepping) using MPI
Use a library such as PETSc or Trilinos. They offer you data structures for parallel linear systems and algorithms that can work on them, such as parallel implementations of CG or GMRES. The PETSc ...
- 50.2k
4
votes
Accepted
Reference Suggestions for MPI
The two books that come to mind are Using MPI and Using Advanced MPI by Gropp and Lusk. They were also released in a 3rd edition last year, so they are very up to date. As others have said, the ...
- 10.8k
4
votes
Accepted
How do I make sparse solvers to accept custom matvec function insted of matrix?
PETSc does this. See the documentation for MatShell and Section 3.3 in the manual.
- 828
4
votes
Accepted
MapReduce with MPI question
As Wolfgang observed, you are basically transposing a connectivity matrix: every process $i$ knows row $i$ that states which $j$ values you're sending to. What you're interested in is the columns: the ...
- 915
4
votes
Accepted
How do I interpret the results from MPI_Wtime()?
That depends on your goal.
For one process it's obvious that $t$ is the wall-time of the entire program launch.
For two processes, you got $t_1$ and $t_2$ which correspond for the walltime on ...
- 8,287
4
votes
Causes for different results for different number of nodes in MPI
There is no way for anyone to tell you where the issue comes from without access to the actual code. But here are common reasons why the output (legitimately) differs for different numbers of ...
- 50.2k
3
votes
Reference Suggestions for MPI
First, you should make sure that you understand the fundamentals of parallel processing first. It's very hard to write good MPI code if you don't understand the principles of distributed computing. ...
- 1,991
3
votes
Is it possible to configure Linux OpenMPI v1.6.5 to use multiple working directories on a single node?
Instead of launching the executable, why don't you launch a script that changes directories into the one you want and then execs your program? OpenMPI sets ...
- 10.8k
3
votes
Any recommendations for unit-testing frameworks compatible with code/libraries that use MPI?
The Teuchos Unit test harness in Trilinos natively supports unit tests that use MPI. Things like controlling output from multiple processes and aggregating pass/fail over all processes is automatic. ...
3
votes
Accepted
How does MPI differentiate between two computers?
The standard is purposefully silent on these issues preferring to leave such issue to library implementers to figure out. There's no need to standardize such mechanisms. Most network hardware has a ...
- 10.8k
3
votes
Accepted
General understanding of Intel MKL, threads and MPI
Nothing stops you from decomposing the problem up yourself and feeding the relevant partitioned data into MKL sequentially, or even in parallel. It will work as long as you avoid data races, but you ...
- 3,203
3
votes
MPI data broadcast or not in C
The question I think you're asking, based on your code, is:
A. Better to have each node do redundant work
B. Better to have one node do the work distribute the results.
The answer is usually A.
To ...
- 3,091
Only top scored, non community-wiki answers of a minimum length are eligible