Possible Duplicate:
What programming paradigms should I be investing in if I want my code to run on petascale machines in the future?
Having entered the multi-core era (som already refers to as the many-core era) for CPUs, I am wondering what would be a good (the best?) future message interface model to be used for writing portable and scalable massively parallel applications on such future many-core CPUs? We can only speculate about it, but, what are the pros/cons?
I think everyone is clear about the fact that MPI can't be the model of the future if we want to use a million-way parallelism. You can read as much in every one of David Keyes' talks, or in any of the many reports on high performance computing over the past decade. It's just that nobody has much else to offer, or just much of an idea how we would structure communication in a better way.
My personal hope is that whatever eventually replaces MPI is at a higher level. MPI is good for sending ints and doubles around, but not for objects. That isn't so difficult, though, as the BOOST MPI bindings show. I'd hope that any more modern replacement would have something like this and doesn't, as MPI has, dabble for 20 years on low-level stuff.
At present, there is no serious contender for distributed memory parallelism, especially when working with libraries. The fact that research languages continue to eschew concepts like communicators and attribute caching (crucial for libraries) suggests that it will continue to dominate for the foreseeable future. My experience has been that MPI consistently performs better and that the MPI Forum has a better understanding of the issues at large scale than any other community.
The case is much less clear for shared memory. On the one hand, memory locality is crucial to get decent performance on systems with a deep NUMA hierarchy. MPI is good for that. On the other hand, the amount of memory per core (or hardware thread) is decreasing rapidly, so strong scalability is becoming more important. The best solution is dependent on the memory hierarchy and details of the job, but it appears that threads accessing shared memory without always making local copies is frequently important. Whether the eventual solution will be OpenCL kernels or more conventional shared memory models has yet to be determined.
I think that for most applications, MPI will continue to be the dominant paradigm for distributed memory. A mostly-separate-memory programming model might be based on future revisions to MPI, but a model with lots of sharing will probably be independent. Library interoperability with threads is an issue that I expect will plague us for at least the next few years.
I think I agree with what most others are saying, which is, roughly:
The thing is, MPI is a really good middleware for some other language to compile down to; but it's shocking that in 2012 we're still expecting our fellow scientists to write raw MPI calls. And I'm not convinced that it's going to get any better; people worry about writing million-way parallelism with MPI, but (a) they don't suggest an alternative, and (b) you wouldn't do million way parallelism with a flat MPI decomposition; you'd program your 128-core nodes with OpenMP (or MPI-3 on-node communicators, or whatever) and use tradaditional MPI to do 8192-way parallelism across nodes, which suddenly seems not all that daunting. Even if you make it 1024-way nodes and 131,072 of them, which gets you up to 130 million, nothing there is clearly impossible.
The problem is that the magical language that allows you to express parallelism so clearly that it can be efficiently parallelized down to distributed memory for you never materialized. Frankly, at this point I think it's pretty clear that it's not going to happen, and the best we can hope for is domain-specific languages; but even there, I dispair that the "domains" there aren't going to be, say "fluid dynamics" but rather "regular-grid finite-difference"; eg, still the wrong level of abstraction.
Some of the recent attempts have been interesting as in "wow, we learned a lot from that effort" but not interesting as in "wow, let's use that for our next project." The people who work with it think Charm++ is great, and the idea seems good - specify the finest level of parallelism and let the runtime worry about assembling that and locality - but it still seems to me like yet another in a series of frameworks that are "perfectly general" as long as you restrict yourself to considering problems that are well-suited to that kind of framework. Stuff like coarray fortran is certainly better than raw mpi, but not enormously -- you still have to decompose the domain and figure out which sub-domain you need to communicate with, you just don't have to write the message-passing yourself. So that's something, but... UPC is interesting, in that in some ways its more ambitious -- you're just supposed to pretend you're writing a serial code and have the array decomposition done for you -- but bolting those global distributed array concepts onto a language that barely has the concept of 1d arrays, and no concept at all of higher-dimensional arrays, was so obviously doomed to failure that you wonder what people were thinking. X10 is just bizarre; how is a program language where you're manually responsible for doing all the forking and joining of threads going to help you with million-way parallelism? Also, java.
The one that strikes me the most interesting right now is Chapel, but that's probably because I'm still trying to find time to play with it. Chapel seems to have really nice mechanisms for separating the code describing the decomposition of the data from the code describing what you do with the data; and it comes with several very common decompositions "built in". It, more than the others, seems like it's approaching the problem the right way around; letting you re-do decompositions without rewriting all of your code, and having several reasonable decompositions baked in.
Last year, David Keyes gave a talk at my school titled "Exaflop/s, Seriously!" When I was searching for the title of the talk, I found that a recorded version of the talk that he gave at another school. Here are the slides from the talk he gave at my school. Here is the paper (without all the pretty graphics) he references in the slides.
My 2 cents MPI will be popluar and used until some one comes up with a simpler solution , a so called parallel language or open source way to automaticlly parallelize serial code.
