# What features do users need from an MPI C++ interface?

The 3.0 version of the MPI standard formally deleted the C++ interface (it was previously deprecated). While implementations may still support it, features that are new in MPI-3 do not have a C++ interface defined in the MPI standard. See http://blogs.cisco.com/performance/the-mpi-c-bindings-what-happened-and-why/ for more information.

The motivation for removing the C++ interface from MPI was that it had no significant value over the C interface. There were very few differences other than "s/_/::/g" and many features that C++ users are accustomed to were not employed (e.g. automatic type determination via templates).

As someone who participates in the MPI Forum and works with a number of C++ projects that have implemented their own C++ interface to the MPI C functions, I would like to know what are the desirable features of a C++ interface to MPI. While I commit to nothing, I would be interested in seeing the implementation of a standalone MPI C++ interface that meets the needs of many users.

And yes, I am familiar with Boost::MPI (http://www.boost.org/doc/libs/1_54_0/doc/html/mpi.html) but it only supports MPI-1 features and the serialization model would be extremely difficult to support for RMA.

One C++ interface to MPI that I like is that of Elemental (https://github.com/poulson/Elemental/blob/master/src/core/imports/mpi.cpp) so perhaps people can provide some pro and con w.r.t. that approach. In particular, I think MpiMap solves an essential problem.

Edit

In response to constructive feedback that this is not appropriate for StackExchange, please move this discussion to MPI Forum issues on GitHub.

• I don't think that this is the appropriate place for such a question. – Jack Poulson Jul 14 '13 at 20:17
• The question is a valuable one, and I think it could get some valuable responses on broader computational science mailing lists, if it's in scope there. (Maybe NA-digest, SIAM-CSE, or even a public post on G+?) This question may not be a good fit for a Stack Exchange site because it is subjective (see scicomp.stackexchange.com/help/dont-ask). As long as answers are concrete and focus on specific use cases (without significant repeats or overlap), I think it's worth keeping open. – Geoff Oxberry Jul 14 '13 at 23:14
• @Jeff: The question comes across much like a poll to me. I don't dispute that it is valuable, but I don't see there being one accepted answer. Would such a question be out-of-the-ordinary for the MPI forum? – Jack Poulson Jul 15 '13 at 0:50
• My concerns echo Jack's. Questions that are answered like polls (see, for instance, Greg Wilson's popular and good question) end up making voting meaningless, and encourages "me too" answers that don't add a whole lot. If the answers start to become like that, then I'll close the question. – Geoff Oxberry Jul 15 '13 at 4:37
• I move to close the issue and have provided a new location on the MPI Forum GitHub to continue the discussion. – Jeff Apr 24 at 14:46

Let me first answer why I think C++ interfaces to MPI have generally not been overly successful, having thought about the issue for a good long time when trying to decide whether we should just use the standard C bindings of MPI or building on something at higher level:

When you look at real-world MPI codes (say, PETSc, or in my case deal.II), one finds that maybe surprisingly, the number of MPI calls isn't actually very large. For example, in the 500k lines of deal.II, there are only ~100 MPI calls. A consequence of this is that the pain involved in using lower-level interfaces such as the MPI C bindings, is not too large. Conversely, one would not gain all that much by using higher level interfaces.

My second observation is that many systems have multiple MPI libraries installed (different MPI implementations, or different versions). This poses a significant difficulty if you wanted to use, say, boost::mpi that don't just consist of header files: either there needs to be multiple installations of this package as well, or one needs to build it as part of the project that uses boost::mpi (but that's a problem in itself again, given that boost uses its own build system, which is unlike anything else).

So I think all of this has conspired against the current crop of C++ interfaces to MPI: The old MPI C++ bindings didn't offer any advantage, and external packages had difficulties with the real world.

This all said, here's what I think would be the killer features I would like to have from a higher-level interface:

• It should be generic. Having to specify the data type of a variable is decidedly not C++-like. Of course, it also leads to errors. Elemental's MpiMap class would already be a nice first step (though I can't figure out why the heck the MpiMap::type variable isn't static const, so that it can be accessed without creating an object).

• It should have facilities for streaming arbitrary data types.

• Operations that require an MPI_Op argument (e.g., reductions) should integrate nicely with C++'s std::function interface, so that it's easy to just pass a function pointer (or a lambda!) rather than having to clumsily register something.

boost::mpi actually satisfies all of these. I think if it were a header-only library, it'd be a lot more popular in practice. It would also help if it supported post-MPI 1.0 functions, but let's be honest: this covers most of what we need most of the time.

• One of the issues with serialization in Boost::MPI is that it doesn't work with one-sided (aka RMA). Do you think that it will be possible to create MPI datatypes for the C++ objects that users are interested in? Of course, in theory all should be supported but I prefer to start with a more pragmatic goal. – Jeff Jul 16 '13 at 14:18
• I think it's a mistake to think that serialized data type may ever work with one-sided communications. This assumes a view that serialization only involves packing data into a string and on the other side unpacking it again. But serialization functions can do much more (e.g., track pointers to other objects, count bytes that have been serialized, etc) than one can expect to work if you can't execute anything on the destination host. My view is that C++-style serialization and one-sided communication is simply a non-starter. I think nobody should expect this to work, so little would be missed. – Wolfgang Bangerth Jul 16 '13 at 17:01
• Hi Wolfgang, you are correct that MpiMap would be more elegant if it used a static const member variable. I have reorganized the implementation: github.com/poulson/Elemental/commit/… – Jack Poulson Jul 16 '13 at 21:52
• Yes, much nicer :-) – Wolfgang Bangerth Jul 17 '13 at 11:00
• +1 about not many mpi calls @WolfgangBangerth . Fundamentally speaking, mpi is supposed to make computations faster, you want to minimize mpi calls because they cost you! – Charles Oct 1 '16 at 1:43

To get the ball rolling, here are two of my needs:

• The interface should be able to eliminate redundant or unnecessary arguments, e.g. MPI_IN_PLACE.
• The interface should auto-detect built-in datatypes ala Elemental's MpiMap.
• If/whenever possible, user-defined datatypes should be constructed for classes.

My list in no particular order of preference. The interface should:

• be header only, without any dependencies but <mpi.h>, and the standard library,
• be generic and extensible,
• be non-blocking only (if you want to block, then block explicitly, not by default),
• allow continuation-based chaining of non-blocking operations,
• support extensible and efficient serialization (Boost.Fusion like, such that it works with RMA),
• have zero abstraction penalty (i.e. be at least as fast as the C interface),
• be safe (the destructor of a non-ready future is called? -> std::terminate!),
• have a strong DEBUG mode with tons of assertions,
• extremely type-safe (no more ints/void* for everything, heck I want tags to be types!),
• it should work with lambdas (e.g. all reduce + lambda),
• use exceptions consistently as error-reporting and error-handling mechanism (no more error codes! no more function output arguments!),
• MPI-IO should offer a non-blocking I/O interface in the style of Boost.AFIO,
• and just follow good modern C++ interface design practices (define regular types, non-member non-friend functions, play well with move semantics, support range operations, ...)

Extras:

• allow me to chose the executor of the MPI environment, that is, which thread pool it uses. Right now you can have applications with a mix of OpenMP, MPI, CUDA, and TBB... all at the same time, where each run-time thinks it owns the environment and thus ask the operating system for threads every time they feel like it. Seriously?

• use the STL (and Boost) naming convention. Why? Every C++ programmer knows it.

I want to write code like this:

auto buffer = some_t{no_ranks};
auto future = gather(comm, root(comm), my_offsets, buffer)
.then([&](){
/* when the gather is finished, this lambda will
execute at the root node, and perform an expensive operation
there asynchronously (compute data required for load
redistribution) whose result is broadcasted to the rest
of the communicator */
}).then([&]() {
/* when broadcast is finished, this lambda executes
on all processes in the communicator, performing an expensive
maybe using non-blocking point-to-point communication) */
return do_something_with(buffer);
}).then([&](auto result) {
/* finally perform a reduction on the result to check
everything went fine */
return all_reduce(comm, root(comm), result,
[](auto acc, auto v) { return acc && v; });
}).then([&](auto result) {
/* check the result at every process */
if (result) { return; /* we are done */ }
else {
root_only([](){ write_some_error_log(); });
throw some_exception;
}
});

/* Here nothing has happened yet! */

/* ... lots and lots of unrelated code that can execute concurrently
and overlaps with communication ... */

/* When we now call future.get() we will block
on the whole chain (which might have finished by then!).
*/

future.get();


Think how one could chain all this operations using MPI_C's requests. You would have to test at multiple (or every single) intermediate step through a whole lot of unrelated code to see if you can advance your chain without blocking.

• MPI can but generally does not use OS threads internally. In every case with which I am familiar (MPICH(2), MVAPICH2, OpenMPI, CrayMPI), only a runtime option provided by the user causes this to occur, i.e. the default is that it does not. Blue Gene/Q MPI is an exception but in such a form as to not be relevant to this discussion. – Jeff Sep 20 '14 at 22:09
• Thanks @Jeff ! Could you elaborate on how MPI handles multiple non-blocking calls while using a single thread? Does it use coroutines/green threads/fibers? – gnzlbg Sep 25 '14 at 6:10
• You must be assuming that MPI is required to make progress on nonblocking operations prior to the subsequent synchronization :-) MPI nonblocking doesn't mean it has to be asynchronous, although good implementations do this and without threads. It is often network hardware that does the asynchronous magic. Honestly, this is quite a long topic and perhaps worthy of a full question on StackOverflow etc. – Jeff Sep 26 '14 at 6:40
• The OS can process some things asynchronously, in which case the dichotomy of hardware or threads is false. And in any case, no threads are spawned in the operation itself - they will be spawned during initialization and managed as efficiently as possible. – Jeff Oct 6 '14 at 5:01

Personally, I don't really mind calling long C-style functions for the exact reason Wolfgang mentioned; there are really few places you need to call them and even then, they almost always get wrapped around by some higher-level code.

The only things that really bother me with C-style MPI are custom datatypes and, to a lesser degree, custom operations (because I use them less often). As for custom datatypes, I'd say that a good C++ interface should be able to support generic and efficient way of handling this, most probably through serialization. This is of course the route that boost.mpi has taken, which if you are careful, is a big time saver.

As for boost.mpi having extra dependencies (particularly boost.serialization which itself is not header-only), I've recently came across a header-only C++ serialization library called cereal which seems promising; granted it requires a C++11 compliant compiler. It might worth looking into and using it as a based for something similar to boost.mpi.

• Note that I was not necessarily looking for something to put in the MPI standard. I completely agree that MPI should nearly always "get wrapped around by some higher-level code", so what I wonder is, what does that higher-level code look like? Elemental has a nice approach, but is it the best for all cases? If one wanted to have a C++ interface to MPI that tried to make a very large number of people happy, what would it look like? – Jeff Oct 20 '13 at 21:10
• @Jeff. For me: 1. I like to be able to send custom data types with ease. 2. I like to be able to perform reduction with custom operations with ease. Also I think 1 i sway more important/useful than 2 – mmirzadeh Nov 13 '13 at 1:41
• I don't see how C++ does anything magical w.r.t. (2). What do you imagine here? – Jeff Nov 14 '13 at 2:16
• @Jeff I was thinking something along the lines of how thrust does it for reductions: docs.thrust.googlecode.com/hg/group__reductions.html – mmirzadeh Nov 14 '13 at 18:50

The github project easyLambda provides a high level interface to MPI with C++14.

I think the project has similar goals and it will give some idea on things that can be and are being done in this area by using modern C++. Guiding other efforts as well as easyLambda itself.

The initial benchmarks on performance and lines of code have shown promising results.

Following is a short description of features and interface it provides.

The interface is based on data flow programming and functional list operations that provide inherent parallelism. The parallelism is expressed as property of a task. The process allocation and data distribution for the task can be requested with a .prll() property. There are good number of examples in the webpage and code-repository that include LAMMPS molecular dynamics post processing, explicit finite difference solution to heat equation, logistic regression etc. As an example the heat diffusion problem discussed in the article HPC is dying... can be expressed in ~20 lines of code.

I hope it is fine to give links rather than adding more details and example codes here.

Disclamer: I am the author of the library. I believe I am not doing any harm in hoping to get a constructive feedback on the current interface of easyLambda that might be advantageous to easyLambda and any other project that pursues similar goals.

• The question says "We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.". Why do you think that your answer is complete enough to fit this description? – nicoguaro Sep 30 '16 at 14:15
• I am pointing towards a project that provides an interface similar to what the OP is asking. I can give details of motivation and features of the project in the answer itself and let the OP and others read and suggest what they think about it. However, I chose to give links only. I get your point. Let me add some text with references to the answer. – Utkarsh Bhardwaj Sep 30 '16 at 22:42
• @UtkarshBhardwaj: Some comments: (1) Thank you for writing a library that interfaces C++ with MPI. It's a significant undertaking and looks like you've put a lot of work into it. (2) Quickly perusing the docs (again, nice job), what stands out is the long chains of method commands used, which seems stylistically...painful to debug, even though you're formatted them nicely. (3) The abstractions presume a functional paradigm, which looks useful for map-reduce-like tasks, but as someone who programs in MPI, I don't want to rework my algorithms and move away too much from the interfaces I know. – Geoff Oxberry Oct 1 '16 at 0:16
• (4) I don't see communicators anywhere in the example, which leads me to suspect you're using MPI_COMM_WORLD for everything, and limits the potential of your library. (5) The abstractions seem to build on the MPI abstractions and could have a different backend. (6) Based on (3)-(5), I don't think this library is an MPI interface, and I believe this comment is off-topic as a result. – Geoff Oxberry Oct 1 '16 at 0:21
• @Geoff thanks for the valuable feedback, much appreciate. Reply to specific points: 2) The chaining is sometimes called as ExpressionBuilder. It is a common way of expressing composition in functional style. It is not necessary to write in this style in ezl. It is possible to write individual units first & then compose them, check the [example] (goo.gl/YzaL0k). 3) I know it is difficult to go from control-flow & imperative to dataflow & functional. Each have there pros & cons. However, I believe the latter needs to be explored more to know its true efficacy. – Utkarsh Bhardwaj Oct 1 '16 at 3:28