It is really impossible to say what the new paradigms will be like in the future, for example a good historical perspective I suggest reading Ken Kennedy's Rise and Fall of HPF. Kennedy gives an account of two emerging patterns, MPI versus a smart compiler, and details how MPI had the right amount of early adopters and flexibility to dominate. HPF eventually fixed its problems but it was too late.
In many ways, several paradigms, such as PGAS and OpenMP, are following that same HPF trend. The early codes have not been flexible enough to use well and left a lot of performance on the table. But the promise of not having to write every iota of the parallel algorithm is a attractive goal. So the pursuit of new models are always being pursued.
Clear Trends in Hardware
Now the success of MPI has often been cited as to being closely tied to how it models the hardware it runs on. Roughly each node has a few number of processes and passing the messages to local point-to-point or through coordinated collective operations is easily done in the cluster space. Because of this, I don't trust anyone who gives a paradigm that doesn't follow closely to new hardware trends, I actually was convinced of this opinion by the work from Vivak Sarakar.
In keeping with that here are three trends that are clearly making headway in new architectures. And let me be clear, there are now twelve different architectures being marketed in HPC. This up from less than 5 years ago only featuring x86, so the coming days will see lots of opportunities for using hardware in different and interesting ways
- Special Purpose Chips: Think large vector units like accelerators (view espoused by Bill Dally of Nvidia)
- Low Power Chips: ARM based clusters (to accomodate power budgets)
- Tiling of Chips: think tiling of chips with different specifications (work of Avant Argwal)
The current model is actually 3 levels deep. While there are many codes using two of these levels well, not many have emerged using all three. I believe that to first get to exascale one needs to invest in determining if you code can run at all three levels. This is probably the safest path for iterating well with the current trends.
Let me iterate on the models and how they will need to change based on the predicted new hardware views.
The players at the distributed level largely fall into MPI and PGAS languages. MPI is a clear winner right now, but PGAS languages such as UPC and Chapel are making headways into the space. One good indication is the HPC Benchmark Challenge. PGAS languages are giving very elegant implementations of the benchmarks.
The most interesting point here is that while this model currently only works at the node level, it will be an important model inside a node for Tiled architectures. One indication is the Intel SCC chip, which fundamentally acted like a distributed system. The SCC team created their own MPI implementation and many teams were successful at porting community libraries to this architecture.
But to be honest PGAS really has a good story for stepping in to this space. Do you really want to program MPI internode and then have to do the same trick intranode? One big deal with these tiled architectures is that they will have different clock speeds on the chips and major differences in bandwidth to memory so performant codes must take this into account.
On-node shared memory
Here we see MPI often being "good enough", but PThreads (and libraries deriving from PThreads such as Intel Parallel Building Blocks) and OpenMP are still used often. The common view is that there will be a time when there are enough shared memory threads that MPI's socket model will break down for RPC or you need a lighter weight process running on the core. Already you can see the indications of IBM Bluegene systems having problems with shared memory MPI.
As Matt comments, the largest performance boost for compute intensive codes is the vectorization of the serial code. While many people assume this is true in accelerators, it is also critical for on-node machines as well. I believe Westmere has a 4 wide FPU, thus one can only get a quarter of the flops without vectorization.
While I don't see the current OpenMP stepping into this space well, there is a place for low-powered or tiles chips to use more light threads. OpenMP has difficulty describing how the data flow works and as more threads are used I only see this trend becoming more exaggerated, just look at examples of what one has to do to get proper prefetching with OpenMP.
Both OpenMP and PThreads at a course enough level can take advantage of the vectorization necessary to get a good percentage of peak, but doing so requires breaking down your algorithms in a way that vectorization is natural.
Finally the emergence of the co-processor (GPU, MIC, Cell acclerators) has taken hold. It is becoming clear that no path to exascale will be complete without them. At SC11, every Bell prize contestent used them very effectively to get to the low petaflops. While CUDA and OpenCL have dominated the current market, I have hopes for OpenACC and PGAS compilers entering the space.
Now to get to exascale, one proposal is to couple the low powered chips to lots of co-processors. This will pretty well kill off the middle layer of the current stack and use codes that manage decision problems on the main chip and shuffle off work to the co-processors. This means that for code to work quite effectively a person must rethink the algorithms in terms of kernels (or codelets), that is branchless instruction level parallel snippets. As far as I know, a solution to this evolution is pretty wide open.
How this affects the app developer
Now to get to your question. If you want to protect yourself from the oncoming complexities of exascale machines, you should do a few things:
- Develop your algorithms to fit at least three levels of parallel hierarchy.
- Design your algorithms in terms of kernels that can be moved between the heirarchy.
- Relax your need for any sequential processes, all of these effects will happen asynchronously because synchronous execution is just not possible.
If you want to be performant today, MPI + CUDA/OpenCL is good enough but UPC is getting there so its not a bad idea to take a few days and learn it. OpenMP gets you started but leads to problems once the code needs to be refactored. PThreads requires completely rewriting your code to its style. Which makes MPI + CUDA/OpenCL the current best model.
What is not discussed here
While all this talk of exascale is nice, something not really discussed here is getting data onto and off of the machines. While there have been many advances in memory systems, we don't see them in commodity cluster (just too darned expensive). Now that data intensive computing is becoming a large focus of all the super computing conferences, there is bound to be a bigger movement into the high memory bandwidth space.
This brings to the other trend that might happen (if the right funding agencies get involved). Machines are going to become more and more specialized for the type of computing required. We already see "data-intensive" machines being funded by the NSF, but these machines are on a different track than the 2019 Exascale Grand Challenge.
This became longer than expected ask for references where you need them in the comments