I have two computers, both with 4-cores and I am running some heavy computational stuff in one of them using Intel Parallel Studio Cluster edition. I am coding in fortran and making extensive use of OpenMP to parallelize the loops.

I read that it should be possible to run this code using the two computers at the same time, but I have no clue on how to set it up. Is it worth it, meaning that, if my code once paralellized on a single machine decreases computational time from 9 hours to 1.37 hours. I guess that if I manage to cluster two machines it will get even faster.

The two machines in which I want to set this up are exactly identical and are connected to the same router via wi fi. My exact question is whether anyone is aware of any good tutorial on how to connect the two machines using MPI? Also, does the coding need to be changed a lot? Is there any good resource on the MPI syntax?



2 Answers 2


Being a big believer in parallel computation (the chips are not getting any faster), the short answer is yes. You should look into MPI (Message Passing Interface), the defecto standard for passing data between different machines when it comes to cluster computing. See this link https://en.wikipedia.org/wiki/Message_Passing_Interface for an intro to MPI.

Assessing whether it is worth it for you to invest time into making the code cluster-safe, to borrow after the thread-safe idiom, depends on how parallelizable your code is, its memory requirements, whether the data is partitioned on a per thread-basis or not, and how much shared data is accessed by each compute node and CPU/core.

That being said, think of MPI as a layer above OpenMP at the level of different computers in your cluster, just as OpenMP which works at the level of different threads (or cores, if you will).


What you mention is called hybrid parallelism.

A cluster is composed of several nodes. A node is a group of sockets that share the same physical memory and a socket is a group of cores that we refer as a processor. The nodes are stored in racks and communicate with each other thanks to a very high-speed connection through a switch.

To simplify, you perform hybrid parallelism when you affect a number of OpenMP threads equal to the number of cores per socket, and a number of MPI processes equal to the number of nodes you want to use on the cluster. For instance, if I have a cluster with 4 nodes and 12 cores by sockets/processors, I launch my program to run with 4 MPI processes (1 by node) and 12 OpenMP threads by MPI process (one by core on each socket). In your example, you have 2 nodes (2 computers, 2 different entities with different memory), and 4 cores by processor. So run with 2 MPI processes, each one dealing with 4 OpenMP threads on each computer. NameRakes gives a good word : MPI is a layer above OpenMP.

An alternative is to only use MPI with 8 processes since MPI works both for shared and distributed memory architecture, you cannot do that with OpenMP which is restricted to shared-memory computers. However, in that case, 8 MPI communications are required against 2 for a hybrid parallelism, which is less efficient.

You seem to have a pretty decent speed-up with OpenMP. Hybrid parallelism makes sense for clusters with dozens of sockets and highly scalable numerical codes with thousand of degrees of freedom. As said by NameRakes, not sure that the time investment will be rewarding since MPI and OpenMP are very different in their concept.

Also be aware that your code will run according to the slower processor so if your two computers have different processors, it may deteriorate your speed-up so you want to make sure to distribute the load correctly. You can surely gain time, but not by a factor 2, it depends on the architecture and the scalability of your code.


Not the answer you're looking for? Browse other questions tagged or ask your own question.