5
$\begingroup$

Possible Duplicate:
What programming paradigms should I be investing in if I want my code to run on petascale machines in the future?

Having entered the multi-core era (som already refers to as the many-core era) for CPUs, I am wondering what would be a good (the best?) future message interface model to be used for writing portable and scalable massively parallel applications on such future many-core CPUs? We can only speculate about it, but, what are the pros/cons?

$\endgroup$

marked as duplicate by Allan P. Engsig-Karup, Aron Ahmadia Feb 23 '12 at 10:54

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 2
    $\begingroup$ This is dangerously close to a double-post. $\endgroup$ – Aron Ahmadia Feb 22 '12 at 20:52
  • $\begingroup$ @aron: I agree. Not clear at the time of writing. I voted this one closed (to be able to keep the answers provided here). $\endgroup$ – Allan P. Engsig-Karup Feb 22 '12 at 22:02
  • $\begingroup$ it's garnered some good responses, so we'll close the question. Anybody interested in responding can go to the thread linked above. $\endgroup$ – Aron Ahmadia Feb 23 '12 at 10:54
6
$\begingroup$

I think everyone is clear about the fact that MPI can't be the model of the future if we want to use a million-way parallelism. You can read as much in every one of David Keyes' talks, or in any of the many reports on high performance computing over the past decade. It's just that nobody has much else to offer, or just much of an idea how we would structure communication in a better way.

My personal hope is that whatever eventually replaces MPI is at a higher level. MPI is good for sending ints and doubles around, but not for objects. That isn't so difficult, though, as the BOOST MPI bindings show. I'd hope that any more modern replacement would have something like this and doesn't, as MPI has, dabble for 20 years on low-level stuff.

$\endgroup$
  • 1
    $\begingroup$ I find it interesting that your suggestion for improving the performance of MPI (the reason MPI can't work with billion-way parallelism is clearly the memory overhead of separate processes on-node, not the programmability) is to provide an abstraction that necessarily involves more packing, more buffering, and more sophisticated handshakes. (Note that MPI demonstrably works for 300k-way parallelism which is pretty close to million-way; but exascale demands billion-way.) Finally, MPI is only intended to be a portable, high-performance transport layer, anything else should be in its own lib. $\endgroup$ – Jed Brown Feb 23 '12 at 2:24
  • 3
    $\begingroup$ I disagree with the trope that MPI cannot handle million-way parallelism. This is flatly wrong (look at the MPICH2 implementation). What is relatedly true is that common distributed memory algorithms start to have real problems on million-way parallel hardware, so I think blaming MPI, and in particular its implementation, is a category error. A good argument for changing to a threading paradigm (I like CUDA) is the ability to keep local memory requirements small, which will be key on future hardware. $\endgroup$ – Matt Knepley Feb 23 '12 at 2:28
  • $\begingroup$ You both misunderstood the direction I was going with my answer. I don't know whether MPI can do a million processes on principle. What I wanted to say is that I think that MPI is the wrong tool for the job. MPI's prevalence has imposed a pretty stiff price on the community in that it has completely permeated the way we think about algorithms. It is, in its design (and, more importantly, in the way most people use it) a library that is by and large synchronous, statically partitioned and requires both side to expect a communication step. It doesn't forgive small imbalances on 1M-way jobs. $\endgroup$ – Wolfgang Bangerth Feb 25 '12 at 8:34
  • 1
    $\begingroup$ What I feel is needed instead is something that supports -- first and foremost, not as an afterthought -- one-sided, asynchronous messaging. Think, for example, Objective-C. And it needs to have better ways of distributing workloads. Think Cilk or a distributed TBB. So what I meant isn't so much that MPI isn't an adequate transport layer (it may be, even for 1M cores) but more that we need new tools to design new kinds of algorithms. I know all of this could be done with MPI, but I'd hope for something that's native to the language and allows to break the mental choke MPI has on us. $\endgroup$ – Wolfgang Bangerth Feb 25 '12 at 8:37
  • $\begingroup$ I've run MPI with 3M processes on Blue Gene/Q and have not found any better tool for scientific applications at this scale. $\endgroup$ – Jeff Dec 10 '14 at 0:17
5
$\begingroup$

At present, there is no serious contender for distributed memory parallelism, especially when working with libraries. The fact that research languages continue to eschew concepts like communicators and attribute caching (crucial for libraries) suggests that it will continue to dominate for the foreseeable future. My experience has been that MPI consistently performs better and that the MPI Forum has a better understanding of the issues at large scale than any other community.

The case is much less clear for shared memory. On the one hand, memory locality is crucial to get decent performance on systems with a deep NUMA hierarchy. MPI is good for that. On the other hand, the amount of memory per core (or hardware thread) is decreasing rapidly, so strong scalability is becoming more important. The best solution is dependent on the memory hierarchy and details of the job, but it appears that threads accessing shared memory without always making local copies is frequently important. Whether the eventual solution will be OpenCL kernels or more conventional shared memory models has yet to be determined.

I think that for most applications, MPI will continue to be the dominant paradigm for distributed memory. A mostly-separate-memory programming model might be based on future revisions to MPI, but a model with lots of sharing will probably be independent. Library interoperability with threads is an issue that I expect will plague us for at least the next few years.

$\endgroup$
5
$\begingroup$

I think I agree with what most others are saying, which is, roughly:

  • Lord, I hope not.
  • But yeah, probably.

The thing is, MPI is a really good middleware for some other language to compile down to; but it's shocking that in 2012 we're still expecting our fellow scientists to write raw MPI calls. And I'm not convinced that it's going to get any better; people worry about writing million-way parallelism with MPI, but (a) they don't suggest an alternative, and (b) you wouldn't do million way parallelism with a flat MPI decomposition; you'd program your 128-core nodes with OpenMP (or MPI-3 on-node communicators, or whatever) and use tradaditional MPI to do 8192-way parallelism across nodes, which suddenly seems not all that daunting. Even if you make it 1024-way nodes and 131,072 of them, which gets you up to 130 million, nothing there is clearly impossible.

The problem is that the magical language that allows you to express parallelism so clearly that it can be efficiently parallelized down to distributed memory for you never materialized. Frankly, at this point I think it's pretty clear that it's not going to happen, and the best we can hope for is domain-specific languages; but even there, I dispair that the "domains" there aren't going to be, say "fluid dynamics" but rather "regular-grid finite-difference"; eg, still the wrong level of abstraction.

Some of the recent attempts have been interesting as in "wow, we learned a lot from that effort" but not interesting as in "wow, let's use that for our next project." The people who work with it think Charm++ is great, and the idea seems good - specify the finest level of parallelism and let the runtime worry about assembling that and locality - but it still seems to me like yet another in a series of frameworks that are "perfectly general" as long as you restrict yourself to considering problems that are well-suited to that kind of framework. Stuff like coarray fortran is certainly better than raw mpi, but not enormously -- you still have to decompose the domain and figure out which sub-domain you need to communicate with, you just don't have to write the message-passing yourself. So that's something, but... UPC is interesting, in that in some ways its more ambitious -- you're just supposed to pretend you're writing a serial code and have the array decomposition done for you -- but bolting those global distributed array concepts onto a language that barely has the concept of 1d arrays, and no concept at all of higher-dimensional arrays, was so obviously doomed to failure that you wonder what people were thinking. X10 is just bizarre; how is a program language where you're manually responsible for doing all the forking and joining of threads going to help you with million-way parallelism? Also, java.

The one that strikes me the most interesting right now is Chapel, but that's probably because I'm still trying to find time to play with it. Chapel seems to have really nice mechanisms for separating the code describing the decomposition of the data from the code describing what you do with the data; and it comes with several very common decompositions "built in". It, more than the others, seems like it's approaching the problem the right way around; letting you re-do decompositions without rewriting all of your code, and having several reasonable decompositions baked in.

$\endgroup$
  • $\begingroup$ You should try Chapel with multi-locale support (or even read the comments about it in the documentation) before suggesting it is a viable solution at scale. $\endgroup$ – Jeff Dec 10 '14 at 0:18
4
$\begingroup$

Last year, David Keyes gave a talk at my school titled "Exaflop/s, Seriously!" When I was searching for the title of the talk, I found that a recorded version of the talk that he gave at another school. Here are the slides from the talk he gave at my school. Here is the paper (without all the pretty graphics) he references in the slides.

$\endgroup$
0
$\begingroup$

My 2 cents MPI will be popluar and used until some one comes up with a simpler solution , a so called parallel language or open source way to automaticlly parallelize serial code.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.