C - OpenMP, MPI, Serial Program

Question

I'm part of a Computational Science course and come from a non-programming background, so please forgive me my ignorance. I'm working on a set of code in C to numerically solve the Navier Stokes equations for a Computational Fluid Dynamics course. As a part of our task, I have to write code that runs on a single processor (serially) as well as provide the choice to parallelize that same code with OpenMP and/or MPI (to utilize multiple processors, depending on user input at the beginning of run time.

I was hoping for advice from the community on doing this and I have a few questions:

Should I be thinking of an if/else structure for my program? (The program would need to branch towards serial code, or MPI enabled code, or OpenMP code, or a hybrid of MPI+OpenMP, and this would mean that much of the code would be duplicated). Should I construct the actions that would be duplicated to be separate functions and call these functions during run time where they are needed? (But from what I understand function calls would increase the overhead of my code in general, regardless of what choice is picked) Would there be a way to hide the pre-processor directives, or MPI-related functions (e.g. MPI_Init()) so that they would conveniently appear or disappear depending on the choice the user made at the beginning or run time? Finally, the purpose of this task is to measure the performance benefits of using serial code, vs. OpenMP, vs. MPI, vs. MPI+OpenMP. Does it make sense that I am trying to avoid code duplication since combining the code (to increase readability) would unnecessarily add overhead to my serial code? Such as MPI-related function calls, potentially many if/else conditions, which would skew the performance measurements of my serial code, versus an if/else structure at the beginning of the program which directs it down a completely separate path (as in Q1).

I hope I've asked a clear question and have abided by the community's standards. Thank you kindly.

(I originally asked this question on what I believe to be the general page of stackoverflow and was directed here.) https://stackoverflow.com/questions/24340121/c-openmp-mpi-serial-program

Answer 1

I think some of your issues are more important than others and some of your emphasis is misplaced. In pursuing overhead, you are in danger of making your program unmaintainable. It is easier to write a common program and direct surplus effort somewhere more interesting. I apologize for pontificating like this.

If statements. From a strict programming perspective, there is no problem having large if-else blocks in your code. If you get your code right, what does it matter? On the other hand, if you don't get your code right there is an issue. With $n$ if-else blocks present, there are something on the order of $2^n$ different paths the execution of your program can take. The choice of parallelism support ($4$ paths) is not the only choice your program will make. These paths can have different outcomes/behaviours (and if you think not, you probably can't be sure), which means you have to make sure each one of those paths is working correctly, which requires a tremendous amount of testing.

So the typical advice is to try and minimize the number of different paths that can be taken through your program. The fewer there are, the fewer need to be checked.

Serial/OMP/MPI/OMP+MPI. Writing four different versions of the same program that are all required to be functionally identical sounds to me like a daft idea. It is not impossible, but why? The extra effort of making sure they all work exactly the same is wasted. Try to maximize the amount of code they share between each other, especially mathematically-tricky code. Spend the extra effort on something interesting instead.

Overhead. In a numerically-intensive program, the overhead of calling a few function is absolutely tiny relative to the computational work, and the overhead of "not needing to communicate" is virtually nil. I would not worry about a few function calls. It would matter in a non-optimized tight inner loop, but for API calls this is not a consideration.

MPI. I don't know how to easily write a program in a way that exactly the same program would easily compile and work both with MPI and without any MPI at all. It is probably far too much effort to try and hide all the MPI calls with ifdefs.

One way is to have two programs, MPI and non-MPI, that rely on the same core mathematical routines. Another way is to make sure the serial version is identical to the MPI version running in a single process. The only overhead here you need to take note of is the overhead of setting up the MPI environment: you need to make sure it's not included in any timing results for the computation, MPI or non-MPI. The overhead of "communicating $0$ bytes" should be nil.

OpenMP. Given how much of OpenMP's parallelism is accessed through compiler pragmas, rather than explicit API calls, many OpenMP programs will compile with and without OpenMP support. It is also probably not worth the effort to hide OpenMP API calls with ifdefs: just run the program with only one thread.

Answer 2

Something in your description sounds amiss when you say you need four codes. You may want clarification from your instructor.

Would one MPI+OpenMP code that produces correct results for any "OMP_NUM_THREADS=n mpiexec -np m" invocation satisfy your assignment? That would permit serial, parallel, multithreaded, and hybrid operation from a single binary at runtime.

Answer 3

Let's say for simplicity that you have a 1D problem, and you discretize it over N grid points. Your code will have a bunch of loops from 1 to N that do stencil operations -- assuming you're not doing implicit time stepping because solving linear systems directly complicates the story considerable.

With this setup, a serial, OpenMP, and MPI version look remarkably similar.

• Your serial version will have the loops from 1 to N.
• Your OpenMP version has the exact same loops, except with a directive that they are parallel.
• Interestingly, the MPI version will still have the same loops, but now your 1..N space is divided over your processors

Ok? So the bulk of your code will look the same in all three cases. Now for the MPI case, which is the hard one. Your big loop gets divided over the processes, so you'd think that one process gets indices 100-200 for instance. Yes, but only in a metaphoric sense. With MPI every process is independent, so it thinks it has addresses 0-100, and you need to translate this back to the global numbering as needed.

The real problem with MPI is that when you apply a stencil you may need some array components that live on another process. Elements 99 and 201 in the above example. That's where you need to do some send and receive operations before you execute your stencil code.

So you have only one code. Can you compile it three ways? Yeah, just sprinkle some #ifdef directives through it. That part is simple.