Skip to main content
Share Your Experience: Take the 2024 Developer Survey
32 votes
Accepted

Why would you need frameworks like MPI when you can multi-task using threads?

There is one real and one practical reason. First, MPI was developed at a time when machines had exactly one processor core and when we wanted to couple different machines. It is today used on ...
Wolfgang Bangerth's user avatar
12 votes
Accepted

What is "good" parallel scaling?

Good is a relative term, and it will depend on the nature of the problem, the nature of the algorithm, and properties of the hardware involved. The only absolute reference point is ideal scaling (100%...
David Ketcheson's user avatar
10 votes
Accepted

Writing a parallel version of an algorithm. Only which someparts are worth distributing

If the operation is as trivial as you say, and each node has all the information necessary to carry out the operation, then the communication will be substantially more expensive than recomputing ...
Daniel Shapero's user avatar
9 votes

Why would you need frameworks like MPI when you can multi-task using threads?

Wolfgang Bangerth's answer is totally correct, and I only want to add one practical aspect. Portability across hardware Let's say you write a research code from scratch. You have a powerful multi-...
MPIchael's user avatar
  • 2,935
6 votes

MPI+OpenMP Scalability

No. You need to test it to that scale, especially if you have MPI calls within OpenMP regions.
Bill Barth's user avatar
  • 10.9k
6 votes

Why my parallel code using MPI is much slower than the serial one?

The first thing you need to ask yourself: is your problem big enough that the overhead of MPI messaging is less than the work that you save. Your problem size is 10k which is small, but on the other ...
Victor Eijkhout's user avatar
5 votes

Distributed (MPI) matrix matrix multiplication

You say that you want an MPI version. Then you need to study the literature, as the distributed memory variant of matrix-matrix product are not a simple parallellization of the sequential version. ...
Victor Eijkhout's user avatar
5 votes

Writing a parallel version of an algorithm. Only which someparts are worth distributing

An addition to @Daniel Shapero's answer. It might also be important to know if there are computations that can lead to different results depending on which machine they are launched (or just vary ...
Anton Menshov's user avatar
  • 8,672
5 votes

CPU usage when a MPI rank waits during a blocking communication

It depends on the communication settings you use for MPI. In blocking communication, MPI has three wait modes. Aggressive busy wait. This is a kind of default mode. Open MPI, at least, uses this when ...
Richard's user avatar
  • 3,971
5 votes

Causes for different results for different number of nodes in MPI

There is no way for anyone to tell you where the issue comes from without access to the actual code. But here are common reasons why the output (legitimately) differs for different numbers of ...
Wolfgang Bangerth's user avatar
4 votes
Accepted

All eigenpairs of large sparse symmetric matrix

To emphasize what the comments have said, if you want to produce the dense 52728x52728 matrix containing the eigenvectors, there is no point to computing using the sparse input. Use a dense input ...
Jeff Hammond's user avatar
  • 2,126
4 votes
Accepted

MapReduce with MPI question

As Wolfgang observed, you are basically transposing a connectivity matrix: every process $i$ knows row $i$ that states which $j$ values you're sending to. What you're interested in is the columns: the ...
Victor Eijkhout's user avatar
4 votes
Accepted

How do I interpret the results from MPI_Wtime()?

That depends on your goal. For one process it's obvious that $t$ is the wall-time of the entire program launch. For two processes, you got $t_1$ and $t_2$ which correspond for the walltime on ...
Anton Menshov's user avatar
  • 8,672
3 votes
Accepted

General understanding of Intel MKL, threads and MPI

Nothing stops you from decomposing the problem up yourself and feeding the relevant partitioned data into MKL sequentially, or even in parallel. It will work as long as you avoid data races, but you ...
Reid.Atcheson's user avatar
3 votes
Accepted

Direct solvers and domain decomposition for FEM

As I understand, this is a fairly popular approach. Direct solvers are usually more efficient than iterative solvers for < 100,000 unknowns, so you partition the problem into subproblems of roughly ...
Daniel Shapero's user avatar
3 votes

MPI data broadcast or not in C

The question I think you're asking, based on your code, is: A. Better to have each node do redundant work B. Better to have one node do the work distribute the results. The answer is usually A. To ...
Richard's user avatar
  • 3,971
3 votes
Accepted

Send Petsc sequential matrix to another MPI rank

I don't think PETSc supports it. PETSc really thinks in parallel, so it converts at most between distributed matrices and sequential matrices through submatrix taking operations. I would MatGetArray ...
Victor Eijkhout's user avatar
3 votes
Accepted

CPU usage when a MPI rank waits during a blocking communication

I can confirm that with blocking waits the receiving processes are spinning at 100 percent. They are probably in a polling loop of sorts. With non-blocking receives you get exactly the same behavior: ...
Victor Eijkhout's user avatar
3 votes

Affecting parts of an array outside MPI process efficiently

The basic model of MPI is "two-sided communication": you have a sender who knows where to send, and a receiver who knows from where to expect something. In your description that is not the ...
Victor Eijkhout's user avatar
3 votes

Recommended CPU for CFD running parallel, FDS (Fire Dynamics Simulator)

Here are some generic recommendations on HPC workstation and server node procurement. First, figure out if you need ECC memory or not, since that determines whether you buy Intel Core and AMD Ryzen ...
Jeff Hammond's user avatar
  • 2,126
3 votes

How does a blocking MPI all to all synchronization work?

Your theory 1 is correct. Why do you think all ranks "release" at the same time? They don't. Why would this even be necessary? Your mental model of MPI is incorrect. Processes run completely ...
Victor Eijkhout's user avatar
3 votes
Accepted

Passing a `MPI_Request*` to the send/recv functions

As mentionned in the comments, the issue was that the request array wasn't big enough. Thus, the first two lines were hiding the memory allocation issue by creating copies of a non-existing ...
Gilles Poncelet's user avatar
3 votes
Accepted

Segmentation fault when broadcasting FFTW wisdom with MPI

The backtrace showed the following: ...
Hannes's user avatar
  • 111
2 votes

Is there an implementation of MPI_AllReduce which handles sparse data better?

I think you may create another vector to store all the non-zero elements in the solution vector. Then use MPI_ALLReduce.
ztdep's user avatar
  • 186
2 votes

How does MPI differentiate between two computers?

Everything Bill Barth wrote is correct. You are confused by MPI processes (or "ranks") and computers. If you do "mpirun -np 18 yourprogram" on a 4-core machine that is not connected to anything, then ...
Victor Eijkhout's user avatar
2 votes
Accepted

MPI support for discrete Fourier transform (DFT) in Python

I think you are looking for mpi4py-fft, which is a Python package (BSD-2 licensed) with its wrappers on the serial FFTW library. From pretty extensive mpi4py documentation: Parallel FFTs are ...
Anton Menshov's user avatar
  • 8,672
2 votes
Accepted

How to determine MPI compatibility

MPI is software that runs on your computer(s). Which implementation and version of MPI you can use therefore depends on the installed operating system. If you are for example running GNU/Linux, you ...
Jannis Teunissen's user avatar
2 votes

MPI in C, all processor run outside of the MPI envir created

Usually, in MPI programs, the MPI_Init(), MPI_Comm_rank and MPI_Comm_size are used only once ...
MakisH's user avatar
  • 151
2 votes

MPI in C, all processor run outside of the MPI envir created

I fear you're confusing your parallel paradigms. MPI supports parallelism with multiple processes (think instances of your program) which communicate with each other across a network. It's possible ...
origimbo's user avatar
  • 2,249
2 votes

MPI+OpenMP Scalability

As a general rule, I would test out the combination of cores and nodes, since scalability depends on so many factors: having enough to do on each node and then core, task / data parallelism in your ...
NameRakes's user avatar
  • 263

Only top scored, non community-wiki answers of a minimum length are eligible