4
$\begingroup$

I am a Mechanical Engineering grad student, currently working on a project which will be scaled up in the new future to require quite some processing power. I am using C++ for the code that I have, and later plan on writing a parallel version of the code for running on distributed memory HPC systems. I do not possess a deep knowledge of the theoretical side of computing, and only know the parallel side of it.

I wanted to ask the community for suggestions on good, easy-to-read-and-understand books or non-video internet resources, which would help me start on parallel programming. Preferably a book which works with plenty of sample C++ codes for each concept that it dictates. I can definitely find more advanced books for myself later, but I would like to get started on my project work with minimum inertia, "in parallel" with my learning.

Do let me know if you know of such resources.

Improve this question
$\endgroup$
5
  • $\begingroup$ I'm not aware of a book, but you can learn it as well by considering code examples for using MPI. Once you understand the principles, you will likely be able to quickly apply it to your code as well. $\endgroup$
    – davidhigh
    Apr 21, 2017 at 10:46
  • $\begingroup$ Would you then know of any good resources online for beginner MPI? I found a course by llnl and argonne national lab, but wanted something more well-structured than them. $\endgroup$
    – Ayush Agrawal
    Apr 21, 2017 at 20:17
  • $\begingroup$ Peter Pacheco has several very readable books on parallel programming including MPI and also several freely-available sets of notes: cs.usfca.edu/~peter/ppmpi $\endgroup$
    – Bill Greene
    Apr 21, 2017 at 23:14
  • $\begingroup$ Well his books are good but none of them directly relate to C++. He also mentions that support for C++ is depreciating, which I have no idea of what it is, and his MPI-2 report that contained the C++ 'bindings' is a dead link, and I could not find a replacement copy on the internet. $\endgroup$
    – Ayush Agrawal
    Apr 22, 2017 at 17:21
  • $\begingroup$ Also look into the petsc library, which can save you a lot of work if your problem fits into what it can do. $\endgroup$
    – cfh
    Apr 24, 2017 at 10:27

3 Answers 3

Reset to default
7
$\begingroup$

One of the first things that you need to understand about parallel programming is the difference between shared memory multiprocessor computer systems and distributed memory clusters.

A shared memory multiprocessor system is a computer in which several processor cores (which might be on one, two, or more integrated circuits) share the same memory. From the programmer's point of view, multiple processes created by the programmer can run on the different cores and share access to the memory. Parallel programs for scientific computing on shared memory systems are typically written using specialized language extensions such as OpenMP (which has C, C++, and Fortran versions.) The OpenMP language extensions make it easy to describe operations on arrays that are to be performed in parallel- the compiler takes care of distributing the work to the multiple processors.

A distributed memory cluster can be thought of as a large collection of computers, each of which has its own memory. In order to interact, processes running on these computers must explicitly send messages to each other. Parallel programs for scientific computing on distributed memory clusters are most commonly written using the message passing interface (MPI) library. Programming with MPI is more difficult than programming with OpenNMP because of the difficulty of deciding how to distribute the work and how processes will communicate by message passing.

Note that a program written using MPI can be run on a shared memory system using a version of the MPI library that simply passes messages between processes through the shared memory. In that sense, an MPI program is much more flexible- it can run on either type of system. In comparison, you can't effectively run an OpenMP program on a distributed memory cluster.

Shared memory systems are typically limited in the number of processor cores and the amount of storage that can be used. In practice, you'll seldom find more than 64 processor cores or about 128 gigabytes of RAM in a shared memory system, while distributed memory clusters might have tens of thousands of processor cores and terabytes of memory.

For truly large scale parallel computing you will need to learn MPI. If you are willing and able to work within the limitations of a smaller shared memory system, than learning OpenMP will be an easier way to get started in parallel computing. For example, if you're using a desktop computer with 4 or 8 cores and you want to take advantage of those cores, then OpenMP is probably the best way to get started.

Other answers have already mentioned some books to get started with MPI. For OpenMP, I'd recommend that you start with the list of resources at

http://www.openmp.org/resources/

Improve this answer
$\endgroup$
3
$\begingroup$

Here's a couple of resources for MPI for C language.

mpitutorial.com

A User's Guide to MPI by Peter Pacheco. This is accessible from the page that @BillGreene referenced in his comment, but this is a convenient PDF while that is a rather inconvenient zipped PostScript.

I think both of these are good resources for beginners.

Regarding C++, it has indeed been deprecated. Here is a discussion. Just stick with the regular C bindings. There's no reason you can't use the regular C calls in a C++ code. If you really want, you can write your own C++ wrapper around the C interface or use Boost MPI, which is a C++ wrapper around MPI.

Improve this answer
$\endgroup$
3
$\begingroup$

I quite liked Parallel Scientific Computing in C++ and MPI when I first used it few years ago. I've a feeling it's C++ is quite dated now and, as has already been observed, MPI and C++ don't play nicely together any more.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.