The 3.0 version of the MPI standard formally deleted the C++ interface (it was previously deprecated). While implementations may still support it, features that are new in MPI-3 do not have a C++ interface defined in the MPI standard. See http://blogs.cisco.com/performance/the-mpi-c-bindings-what-happened-and-why/ for more information.

The motivation for removing the C++ interface from MPI was that it had no significant value over the C interface. There were very few differences other than "s/_/::/g" and many features that C++ users are accustomed to were not employed (e.g. automatic type determination via templates).

As someone who participates in the MPI Forum and works with a number of C++ projects that have implemented their own C++ interface to the MPI C functions, I would like to know what are the desirable features of a C++ interface to MPI. While I commit to nothing, I would be interested in seeing the implementation of a standalone MPI C++ interface that meets the needs of many users.

And yes, I am familiar with Boost::MPI (http://www.boost.org/doc/libs/1_54_0/doc/html/mpi.html) but it only supports MPI-1 features and the serialization model would be extremely difficult to support for RMA.

One C++ interface to MPI that I like is that of Elemental (https://github.com/poulson/Elemental/blob/master/src/core/imports/mpi.cpp) so perhaps people can provide some pro and con w.r.t. that approach. In particular, I think MpiMap solves an essential problem.


In response to constructive feedback that this is not appropriate for StackExchange, please move this discussion to MPI Forum issues on GitHub.

  • $\begingroup$ I don't think that this is the appropriate place for such a question. $\endgroup$ Commented Jul 14, 2013 at 20:17
  • 1
    $\begingroup$ The question is a valuable one, and I think it could get some valuable responses on broader computational science mailing lists, if it's in scope there. (Maybe NA-digest, SIAM-CSE, or even a public post on G+?) This question may not be a good fit for a Stack Exchange site because it is subjective (see scicomp.stackexchange.com/help/dont-ask). As long as answers are concrete and focus on specific use cases (without significant repeats or overlap), I think it's worth keeping open. $\endgroup$ Commented Jul 14, 2013 at 23:14
  • 1
    $\begingroup$ @Jeff: The question comes across much like a poll to me. I don't dispute that it is valuable, but I don't see there being one accepted answer. Would such a question be out-of-the-ordinary for the MPI forum? $\endgroup$ Commented Jul 15, 2013 at 0:50
  • 1
    $\begingroup$ My concerns echo Jack's. Questions that are answered like polls (see, for instance, Greg Wilson's popular and good question) end up making voting meaningless, and encourages "me too" answers that don't add a whole lot. If the answers start to become like that, then I'll close the question. $\endgroup$ Commented Jul 15, 2013 at 4:37
  • 1
    $\begingroup$ I move to close the issue and have provided a new location on the MPI Forum GitHub to continue the discussion. $\endgroup$ Commented Apr 24, 2020 at 14:46

5 Answers 5


Let me first answer why I think C++ interfaces to MPI have generally not been overly successful, having thought about the issue for a good long time when trying to decide whether we should just use the standard C bindings of MPI or building on something at higher level:

When you look at real-world MPI codes (say, PETSc, or in my case deal.II), one finds that maybe surprisingly, the number of MPI calls isn't actually very large. For example, in the 500k lines of deal.II, there are only ~100 MPI calls. A consequence of this is that the pain involved in using lower-level interfaces such as the MPI C bindings, is not too large. Conversely, one would not gain all that much by using higher level interfaces.

My second observation is that many systems have multiple MPI libraries installed (different MPI implementations, or different versions). This poses a significant difficulty if you wanted to use, say, boost::mpi that don't just consist of header files: either there needs to be multiple installations of this package as well, or one needs to build it as part of the project that uses boost::mpi (but that's a problem in itself again, given that boost uses its own build system, which is unlike anything else).

So I think all of this has conspired against the current crop of C++ interfaces to MPI: The old MPI C++ bindings didn't offer any advantage, and external packages had difficulties with the real world.

This all said, here's what I think would be the killer features I would like to have from a higher-level interface:

  • It should be generic. Having to specify the data type of a variable is decidedly not C++-like. Of course, it also leads to errors. Elemental's MpiMap class would already be a nice first step (though I can't figure out why the heck the MpiMap::type variable isn't static const, so that it can be accessed without creating an object).

  • It should have facilities for streaming arbitrary data types.

  • Operations that require an MPI_Op argument (e.g., reductions) should integrate nicely with C++'s std::function interface, so that it's easy to just pass a function pointer (or a lambda!) rather than having to clumsily register something.

boost::mpi actually satisfies all of these. I think if it were a header-only library, it'd be a lot more popular in practice. It would also help if it supported post-MPI 1.0 functions, but let's be honest: this covers most of what we need most of the time.

  • 1
    $\begingroup$ One of the issues with serialization in Boost::MPI is that it doesn't work with one-sided (aka RMA). Do you think that it will be possible to create MPI datatypes for the C++ objects that users are interested in? Of course, in theory all should be supported but I prefer to start with a more pragmatic goal. $\endgroup$ Commented Jul 16, 2013 at 14:18
  • 1
    $\begingroup$ I think it's a mistake to think that serialized data type may ever work with one-sided communications. This assumes a view that serialization only involves packing data into a string and on the other side unpacking it again. But serialization functions can do much more (e.g., track pointers to other objects, count bytes that have been serialized, etc) than one can expect to work if you can't execute anything on the destination host. My view is that C++-style serialization and one-sided communication is simply a non-starter. I think nobody should expect this to work, so little would be missed. $\endgroup$ Commented Jul 16, 2013 at 17:01
  • 1
    $\begingroup$ Hi Wolfgang, you are correct that MpiMap would be more elegant if it used a static const member variable. I have reorganized the implementation: github.com/poulson/Elemental/commit/… $\endgroup$ Commented Jul 16, 2013 at 21:52
  • $\begingroup$ Yes, much nicer :-) $\endgroup$ Commented Jul 17, 2013 at 11:00
  • 1
    $\begingroup$ +1 about not many mpi calls @WolfgangBangerth . Fundamentally speaking, mpi is supposed to make computations faster, you want to minimize mpi calls because they cost you! $\endgroup$
    – Charles
    Commented Oct 1, 2016 at 1:43

To get the ball rolling, here are two of my needs:

  • The interface should be able to eliminate redundant or unnecessary arguments, e.g. MPI_IN_PLACE.
  • The interface should auto-detect built-in datatypes ala Elemental's MpiMap.
  • If/whenever possible, user-defined datatypes should be constructed for classes.

My list in no particular order of preference. The interface should:

  • be header only, without any dependencies but <mpi.h>, and the standard library,
  • be generic and extensible,
  • be non-blocking only (if you want to block, then block explicitly, not by default),
  • allow continuation-based chaining of non-blocking operations,
  • support extensible and efficient serialization (Boost.Fusion like, such that it works with RMA),
  • have zero abstraction penalty (i.e. be at least as fast as the C interface),
  • be safe (the destructor of a non-ready future is called? -> std::terminate!),
  • have a strong DEBUG mode with tons of assertions,
  • extremely type-safe (no more ints/void* for everything, heck I want tags to be types!),
  • it should work with lambdas (e.g. all reduce + lambda),
  • use exceptions consistently as error-reporting and error-handling mechanism (no more error codes! no more function output arguments!),
  • MPI-IO should offer a non-blocking I/O interface in the style of Boost.AFIO,
  • and just follow good modern C++ interface design practices (define regular types, non-member non-friend functions, play well with move semantics, support range operations, ...)


  • allow me to chose the executor of the MPI environment, that is, which thread pool it uses. Right now you can have applications with a mix of OpenMP, MPI, CUDA, and TBB... all at the same time, where each run-time thinks it owns the environment and thus ask the operating system for threads every time they feel like it. Seriously?

  • use the STL (and Boost) naming convention. Why? Every C++ programmer knows it.

I want to write code like this:

auto buffer = some_t{no_ranks};
auto future = gather(comm, root(comm), my_offsets, buffer)
                /* when the gather is finished, this lambda will 
                   execute at the root node, and perform an expensive operation
                   there asynchronously (compute data required for load 
                   redistribution) whose result is broadcasted to the rest 
                   of the communicator */
                return broadcast(comm, root(comm), buffer);
              }).then([&]() {
                /* when broadcast is finished, this lambda executes 
                   on all processes in the communicator, performing an expensive
                   operation asynchronously (redistribute the load, 
                   maybe using non-blocking point-to-point communication) */
                 return do_something_with(buffer);
              }).then([&](auto result) {
                 /* finally perform a reduction on the result to check
                    everything went fine */
                 return all_reduce(comm, root(comm), result, 
                                  [](auto acc, auto v) { return acc && v; }); 
              }).then([&](auto result) {
                  /* check the result at every process */
                  if (result) { return; /* we are done */ }
                  else {
                    root_only([](){ write_some_error_log(); });
                    throw some_exception;

/* Here nothing has happened yet! */

/* ... lots and lots of unrelated code that can execute concurrently 
   and overlaps with communication ... */

/* When we now call future.get() we will block 
   on the whole chain (which might have finished by then!).


Think how one could chain all this operations using MPI_C's requests. You would have to test at multiple (or every single) intermediate step through a whole lot of unrelated code to see if you can advance your chain without blocking.

  • 2
    $\begingroup$ MPI can but generally does not use OS threads internally. In every case with which I am familiar (MPICH(2), MVAPICH2, OpenMPI, CrayMPI), only a runtime option provided by the user causes this to occur, i.e. the default is that it does not. Blue Gene/Q MPI is an exception but in such a form as to not be relevant to this discussion. $\endgroup$ Commented Sep 20, 2014 at 22:09
  • 1
    $\begingroup$ Thanks @Jeff ! Could you elaborate on how MPI handles multiple non-blocking calls while using a single thread? Does it use coroutines/green threads/fibers? $\endgroup$
    – gnzlbg
    Commented Sep 25, 2014 at 6:10
  • 1
    $\begingroup$ You must be assuming that MPI is required to make progress on nonblocking operations prior to the subsequent synchronization :-) MPI nonblocking doesn't mean it has to be asynchronous, although good implementations do this and without threads. It is often network hardware that does the asynchronous magic. Honestly, this is quite a long topic and perhaps worthy of a full question on StackOverflow etc. $\endgroup$ Commented Sep 26, 2014 at 6:40
  • 1
    $\begingroup$ @Jeff Thanks I didnt thought about offloading. If there is hardware offloading then yes the call to a non-blocking operation could be just offloaded to the hardware and the test/wait calls just check if the operation completed (no threads required). Without hardware offloading, however, the implementation either blocks, or has to spawn a thread. If you could point me to some literature I would be terribly in doubt with you. It is a very interesting topic. I really don't know how much part of the MPI calls can be completely offloaded to the hardware besides send and recv. $\endgroup$
    – gnzlbg
    Commented Sep 26, 2014 at 17:12
  • 1
    $\begingroup$ The OS can process some things asynchronously, in which case the dichotomy of hardware or threads is false. And in any case, no threads are spawned in the operation itself - they will be spawned during initialization and managed as efficiently as possible. $\endgroup$ Commented Oct 6, 2014 at 5:01

Personally, I don't really mind calling long C-style functions for the exact reason Wolfgang mentioned; there are really few places you need to call them and even then, they almost always get wrapped around by some higher-level code.

The only things that really bother me with C-style MPI are custom datatypes and, to a lesser degree, custom operations (because I use them less often). As for custom datatypes, I'd say that a good C++ interface should be able to support generic and efficient way of handling this, most probably through serialization. This is of course the route that boost.mpi has taken, which if you are careful, is a big time saver.

As for boost.mpi having extra dependencies (particularly boost.serialization which itself is not header-only), I've recently came across a header-only C++ serialization library called cereal which seems promising; granted it requires a C++11 compliant compiler. It might worth looking into and using it as a based for something similar to boost.mpi.

  • $\begingroup$ Note that I was not necessarily looking for something to put in the MPI standard. I completely agree that MPI should nearly always "get wrapped around by some higher-level code", so what I wonder is, what does that higher-level code look like? Elemental has a nice approach, but is it the best for all cases? If one wanted to have a C++ interface to MPI that tried to make a very large number of people happy, what would it look like? $\endgroup$ Commented Oct 20, 2013 at 21:10
  • $\begingroup$ @Jeff. For me: 1. I like to be able to send custom data types with ease. 2. I like to be able to perform reduction with custom operations with ease. Also I think 1 i sway more important/useful than 2 $\endgroup$
    – mmirzadeh
    Commented Nov 13, 2013 at 1:41
  • $\begingroup$ I don't see how C++ does anything magical w.r.t. (2). What do you imagine here? $\endgroup$ Commented Nov 14, 2013 at 2:16
  • $\begingroup$ @Jeff I was thinking something along the lines of how thrust does it for reductions: docs.thrust.googlecode.com/hg/group__reductions.html $\endgroup$
    – mmirzadeh
    Commented Nov 14, 2013 at 18:50

The github project easyLambda provides a high level interface to MPI with C++14.

I think the project has similar goals and it will give some idea on things that can be and are being done in this area by using modern C++. Guiding other efforts as well as easyLambda itself.

The initial benchmarks on performance and lines of code have shown promising results.

enter image description here

Following is a short description of features and interface it provides.

The interface is based on data flow programming and functional list operations that provide inherent parallelism. The parallelism is expressed as property of a task. The process allocation and data distribution for the task can be requested with a .prll() property. There are good number of examples in the webpage and code-repository that include LAMMPS molecular dynamics post processing, explicit finite difference solution to heat equation, logistic regression etc. As an example the heat diffusion problem discussed in the article HPC is dying... can be expressed in ~20 lines of code.

I hope it is fine to give links rather than adding more details and example codes here.

Disclamer: I am the author of the library. I believe I am not doing any harm in hoping to get a constructive feedback on the current interface of easyLambda that might be advantageous to easyLambda and any other project that pursues similar goals.

  • $\begingroup$ The question says "We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.". Why do you think that your answer is complete enough to fit this description? $\endgroup$
    – nicoguaro
    Commented Sep 30, 2016 at 14:15
  • $\begingroup$ I am pointing towards a project that provides an interface similar to what the OP is asking. I can give details of motivation and features of the project in the answer itself and let the OP and others read and suggest what they think about it. However, I chose to give links only. I get your point. Let me add some text with references to the answer. $\endgroup$ Commented Sep 30, 2016 at 22:42
  • $\begingroup$ @UtkarshBhardwaj: Some comments: (1) Thank you for writing a library that interfaces C++ with MPI. It's a significant undertaking and looks like you've put a lot of work into it. (2) Quickly perusing the docs (again, nice job), what stands out is the long chains of method commands used, which seems stylistically...painful to debug, even though you're formatted them nicely. (3) The abstractions presume a functional paradigm, which looks useful for map-reduce-like tasks, but as someone who programs in MPI, I don't want to rework my algorithms and move away too much from the interfaces I know. $\endgroup$ Commented Oct 1, 2016 at 0:16
  • $\begingroup$ (4) I don't see communicators anywhere in the example, which leads me to suspect you're using MPI_COMM_WORLD for everything, and limits the potential of your library. (5) The abstractions seem to build on the MPI abstractions and could have a different backend. (6) Based on (3)-(5), I don't think this library is an MPI interface, and I believe this comment is off-topic as a result. $\endgroup$ Commented Oct 1, 2016 at 0:21
  • $\begingroup$ @Geoff thanks for the valuable feedback, much appreciate. Reply to specific points: 2) The chaining is sometimes called as ExpressionBuilder. It is a common way of expressing composition in functional style. It is not necessary to write in this style in ezl. It is possible to write individual units first & then compose them, check the [example] (goo.gl/YzaL0k). 3) I know it is difficult to go from control-flow & imperative to dataflow & functional. Each have there pros & cons. However, I believe the latter needs to be explored more to know its true efficacy. $\endgroup$ Commented Oct 1, 2016 at 3:28

Not the answer you're looking for? Browse other questions tagged or ask your own question.