14
$\begingroup$

MPI is an interface which enables us to create multiple processes to be run on a single machine or on a cluster of machines, and enables message passing or in short sorts of communication between processes.

I am sure they are other lots of specifications which enables multi processing to execute one bigger task. However, multi tasking and breaking a bigger task into smaller can also be done via threads. As far as I understand creating threads is much faster compared to processes and it does not need any message passing to communicate as shared memory is inherent.

Why do you even need specifications like MPI and others for multi processing when you can achieve same effect using multi threaded programs ?

Improve this question
$\endgroup$
6
  • 22
    $\begingroup$ Your question starts with a false premise. Threading with shared memory isn't possible across a cluster of machines. $\endgroup$
    – Brian Borchers
    Apr 22, 2020 at 14:03
  • 2
    $\begingroup$ I would say that MPI can be faster as you can create multiple processes instead of multiple threads. Multiple processes can work on their own little piece of the problem independent of how far the other processes are. But the communication overhead for MPI is higher as you need to distribute the data and get it back to the parent process at the end. $\endgroup$
    – vydesaster
    Apr 22, 2020 at 14:15
  • $\begingroup$ @vydesaster There's no way creating a process is faster than a thread. Performance-wise threads will beat processes every day of the week. There are a lot of downsides to threads though. $\endgroup$
    – Voo
    Apr 24, 2020 at 8:00
  • $\begingroup$ @vydesaster Not necessarily "back to the parent process", but rather out of the application. That can also be done by several processes writing parts of the data to independent files on a parallel file system. The data will then only be fully joined together within some post-processing app, if ever. $\endgroup$
    – cmaster - reinstate monica
    Apr 24, 2020 at 9:46
  • $\begingroup$ @voo Actually in the HPC world multiple process, single threaded programs generally outperform multiple process, multiple threaded programs up to a smallish number of cores (say a few 100 but it is very application dependent), when multiple processes with a small number threads becomes quicker. Threads generally require more synchronisation than processes due to the possibility of shared resources, and this can slow such codes down compared to a process implementation where there is (usually) no sharing. But it is very case dependent. $\endgroup$
    – Ian Bush
    Apr 24, 2020 at 15:25

3 Answers 3

Reset to default
32
$\begingroup$

There is one real and one practical reason.

First, MPI was developed at a time when machines had exactly one processor core and when we wanted to couple different machines. It is today used on clusters of tens of thousands of machines, each of which happens to have many cores but the point is that it's still separate machines. Now, a processor core on machine A can't access memory on machine B, and so there needs to be a way to transfer information between these processes -- that's what the message passing interface (MPI) fundamentally does: transfer data from one machine to another.

You are entirely correct that, strictly speaking, you don't need MPI if you are working on one machine only. That of course limits how far you can scale your program (you will be able to use a few dozen threads, but not thousands since we don't have machines with that many cores). But more importantly, when you use threads, you now have a few dozen threads all accessing the same memory. It turns out to be conceptually very difficult to write codes that are efficient because historically we have been taught that the way to access shared data structures is to just use a mutex to access the information. That turns out to be efficient if you have 4 cores access the same memory, but not if you have 192: In that case, the ratio of time spent on computing information to time spent obtaining the mutex is just not very good any more. What one needs to do to address the issue is that every thread duplicates the read-write data structures during the main phase of the algorithm (so that they can be accessed without a mutex), followed by a reduction step. In other words, threads need to keep separate copies of data structures for efficiency. But that's not how we think when we program with threads, and so few implementations employ this strategy. On the other hand, that's what you need to do when you program with MPI because every process has its own memory space -- so MPI forces you to do what you should do with threads, and that's why using MPI often leads to quite efficient and scalable programs even when used in situations where threads could be used.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ If you're willing to deal with the inability to scale to a larger cluster, shared memory programming using OpenMP or OpenACC is an alternative that may be easier to get into than using MPI and that avoids many of the difficulties of programming with threads. $\endgroup$
    – Brian Borchers
    Apr 22, 2020 at 17:01
  • $\begingroup$ Yes. Or even better use modern C++ facilities where you can use tasking and parallel for_each statements applied to iterator ranges. The point I'm trying to make in my last paragraph stands, however: It turns out to be very difficult to produce really efficient codes if your mindset is on shared-memory threads. You need to treat threads as distributed-memory approaches for many algorithms is you really want to scale well. $\endgroup$
    – Wolfgang Bangerth
    Apr 23, 2020 at 2:42
  • $\begingroup$ The question seems to be better, if asking why would you need MPI when you can use sockets? $\endgroup$
    – Tim
    Apr 24, 2020 at 0:21
  • 1
    $\begingroup$ @Tim: Why would you need sockets if you could just send a TCP packet? But then also why use TCP if you could use UDP! $\endgroup$
    – Wolfgang Bangerth
    Apr 25, 2020 at 3:49
  • $\begingroup$ What do you mean by "Why would you need sockets if you could just send a TCP packet? But then also why use TCP if you could use UDP"? $\endgroup$
    – Tim
    Apr 25, 2020 at 8:17
9
$\begingroup$

Wolfgang Bangerth's answer is totally correct, and I only want to add one practical aspect.

Portability across hardware

Let's say you write a research code from scratch. You have a powerful multi-core shared memory machine at your department which can do the job. If you start out with a thread-based implementation, you can reach good performance on this particular machine.

That is fine as long as you have a fixed, and well-defined problem description. However, in reality, every research question you ask will lead to more questions. Soon you need to do a parameter study that multiplies your numerical demand and you outgrow your accessible hardware.

If you start out with MPI from the beginning, you have fewer problems to scale your problem upon another cluster and you are not limited to one particular architecture. You can spin up your code on your desktop, on the multicore SMD machine, and on a university cluster all the same!

Improve this answer
$\endgroup$
6
  • 2
    $\begingroup$ This isn't really such a good argument anymore. There are now a lot of situations where MPI does not do the job. In particular, MPI does not help with GPUs, nor is it very practical for more asynchronous/concurrent/decentralised setups. Of course threads as such don't help much either, but the point is: to get real portability, both MPI and threads are simply too low-level abstractions. $\endgroup$
    – leftaroundabout
    Apr 23, 2020 at 10:32
  • $\begingroup$ Well. I would not want to chain myself to GPU's or any co-processor architecture. In my (limited) experience, any University cluster will have a small number of nodes which have graphics cards in them. If your problem fits into these specifications then that is fine. But with co-processors the on- and offloading times limit your scaling significantly if you have to communicate. If you go with cpu's and distributed memory you can scale a lot further and on any cluster. That's just my two cents $\endgroup$
    – MPIchael
    Apr 23, 2020 at 13:10
  • $\begingroup$ I might as well make that argument for “wouldn't want to chain myself to a cluster-of-CPUs architecture”. For some tasks a single GPU is sufficient or else 100 CPU cores, and then the GPU is more economical and much more convenient. $\endgroup$
    – leftaroundabout
    Apr 23, 2020 at 14:03
  • $\begingroup$ If you want your job to run as fast as possible then you know what machine you have and you do want to tailor it to that machine. Especially if it's research code which probably runs once or several times on the same machine and then is never touched again. $\endgroup$
    – user253751
    Apr 23, 2020 at 14:57
  • 1
    $\begingroup$ @leftaroundabout, “both MPI and threads are simply too low-level abstractions” -- absolutely. MPI certainly shows its age. There are frameworks for specific applications, like TensorFlow 2 for large ML tasks, which introduce abstractions for intra-cluster strategies. This is a step in the right direction. But I'd say it's not comparing apples to apples. I'm yet to see an attempt at a less task-specific higher-level distributed abstraction. E.g., Intel TBB and similar frameworks with a Graph abstraction is one step above threads. But nothing like that one level above MPI exists yet, AFAIK. $\endgroup$
    – kkm -still wary of SE promises
    Apr 25, 2020 at 8:36
2
$\begingroup$

MPI solves a different problem than multithreading, whether it's done via pthreads or OpenMP:

  • Multithreading is designed to take advantage of a single, big machine, but is restricted to that one machine. If you server only has 64 processor cores, that's the max. amount of threads that can be run (if you care for performance, that is).

  • MPI is designed to scale an applications beyond that single machine. You can use as many machines as your money can buy, plug them together with some (high-performance) network, and you can execute an MPI application that utilizes your entire, multi-million dollar supercomputer at once.

Climate simulations, weather forecasts, physics simulations, etc. are frequently run in such a massively parallel fashion using MPI. Not because everybody likes using MPI, but rather because you simply cannot run a high-resolution climate model on a single machine anymore. That would take too long, and you simply can't put enough RAM into a single machine for that.

These are the advantages of using MPI over OpenMP or pthreads:

  • Security: Often forgotten, but you cannot produce a data race if you have no shared data that you could race on. Processes don't share data, so you can completely forget about grabbing locks, etc. when programming for MPI. This makes reasoning about your source code much simpler. All communication is explicit and any receive will block until it's safe to continue.

    Note that the modern MPI-standarts have been putting water in this wine for performance reasons: They include non-blocking send/receive calls, and primitives for remote memory access. These open explicit breaches in the memory isolation of the process. Nevertheless, even with these less safe communications, it is very clear where data races are possible, and where not. All the holes are very explicit, and limited in time. That's still very different from the multithreading world where you basically need to ask yourself whether you need to grab a lock whenever you touch a memory object.

  • Performance Scaling: You are not restricted to the count of CPU cores on a single machine.

  • Memory Scaling: There is a maximum of RAM that you can plug into a single machine. With MPI, you can scatter your data across many smaller machines. Today, this may be more important than performance scaling!

Finally, using MPI does not mean that you cannot use OpenMP on top. This can be done, and it is done: You start a low amount of processes per machine, and each process uses OpenMP to utilize the available CPU cores. This is called hybrid parallelization. It significantly reduces the amount of data that needs to be communicated via MPI (improves performance), with the tradeoff of making the code more complex.

Improve this answer
$\endgroup$
4
  • $\begingroup$ These are all good arguments, though it's worth pointing that if you use blocking send and receive operations in MPI, then you're often doing something wrong :-) The best performing algorithms are those where you can interleave computations and communication by way of non-blocking calls, and then race conditions can very much happen. Though it's much less common to have those in your code. $\endgroup$
    – Wolfgang Bangerth
    Apr 25, 2020 at 3:53
  • $\begingroup$ @WolfgangBangerth Yes, there are non-blocking APIs in the current standard. Two classes, in fact: Immediate send/receive, and remote memory access. Obviously, the remote memory access does destroy the security advantage for the data within the MPI-window(s). On the other hand, an immediate send/receive is not complete until the corresponding MPI_Wait() has been called, so it's more like an explicitly smeared out operation than true concurrency. And once the operation is complete, we are back to being data-race free. But I agree, I should probably add a note on this in my answer. $\endgroup$
    – cmaster - reinstate monica
    Apr 25, 2020 at 7:59
  • $\begingroup$ @WolfgangBangerth Ok, I have now added a note on data-racy MPI primitives. I hope it finds your consent. $\endgroup$
    – cmaster - reinstate monica
    Apr 25, 2020 at 8:11
  • $\begingroup$ It does :-) You're right that in practice, the fact that one should use MPI_Wait() makes sure that it is not common to write codes that have data races. $\endgroup$
    – Wolfgang Bangerth
    Apr 25, 2020 at 14:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.