# C - OpenMP, MPI, Serial Program

I'm part of a Computational Science course and come from a non-programming background, so please forgive me my ignorance. I'm working on a set of code in C to numerically solve the Navier Stokes equations for a Computational Fluid Dynamics course. As a part of our task, I have to write code that runs on a single processor (serially) as well as provide the choice to parallelize that same code with OpenMP and/or MPI (to utilize multiple processors, depending on user input at the beginning of run time.

I was hoping for advice from the community on doing this and I have a few questions:

Should I be thinking of an if/else structure for my program? (The program would need to branch towards serial code, or MPI enabled code, or OpenMP code, or a hybrid of MPI+OpenMP, and this would mean that much of the code would be duplicated). Should I construct the actions that would be duplicated to be separate functions and call these functions during run time where they are needed? (But from what I understand function calls would increase the overhead of my code in general, regardless of what choice is picked) Would there be a way to hide the pre-processor directives, or MPI-related functions (e.g. MPI_Init()) so that they would conveniently appear or disappear depending on the choice the user made at the beginning or run time? Finally, the purpose of this task is to measure the performance benefits of using serial code, vs. OpenMP, vs. MPI, vs. MPI+OpenMP. Does it make sense that I am trying to avoid code duplication since combining the code (to increase readability) would unnecessarily add overhead to my serial code? Such as MPI-related function calls, potentially many if/else conditions, which would skew the performance measurements of my serial code, versus an if/else structure at the beginning of the program which directs it down a completely separate path (as in Q1).

I hope I've asked a clear question and have abided by the community's standards. Thank you kindly.

(I originally asked this question on what I believe to be the general page of stackoverflow and was directed here.) https://stackoverflow.com/questions/24340121/c-openmp-mpi-serial-program

• Thank you all for your responses, they've been most valuable. I was also more interested in minimizing the amount of repeated work, but was just worried that having MPI/OpenMP related code in place (during the serial operation - when threads and procs are set to 1) would cause an overhead to what is supposed to be "serial" code and would skew perceived performance improvements when examining parallel code speedup. I suppose with your responses, that overhead is negligible and I can simply set flags to 1 (to run serially) instead of duplicating code multiple times, which indeed is daft. – kfkhalili Jun 24 '14 at 11:45
• Hey. The nice thing about precompiler if-defs is, that effectively you -do- have different codes. Think about the ifdefs as one path of the precompiler where only the relevant code pieces are left inside the code. Then on a second sweep the actual compilation happens. That means you don't have to worry about interference or performance drops of the mpi/openMP stuff, if it is not switched on. It simply doesn't make it into the executable! You can check the precompiler output via a flag to the compiler. – MPIchael Aug 26 '19 at 8:30

I think some of your issues are more important than others and some of your emphasis is misplaced. In pursuing overhead, you are in danger of making your program unmaintainable. It is easier to write a common program and direct surplus effort somewhere more interesting. I apologize for pontificating like this.

If statements. From a strict programming perspective, there is no problem having large if-else blocks in your code. If you get your code right, what does it matter? On the other hand, if you don't get your code right there is an issue. With $n$ if-else blocks present, there are something on the order of $2^n$ different paths the execution of your program can take. The choice of parallelism support ($4$ paths) is not the only choice your program will make. These paths can have different outcomes/behaviours (and if you think not, you probably can't be sure), which means you have to make sure each one of those paths is working correctly, which requires a tremendous amount of testing.

So the typical advice is to try and minimize the number of different paths that can be taken through your program. The fewer there are, the fewer need to be checked.

Serial/OMP/MPI/OMP+MPI. Writing four different versions of the same program that are all required to be functionally identical sounds to me like a daft idea. It is not impossible, but why? The extra effort of making sure they all work exactly the same is wasted. Try to maximize the amount of code they share between each other, especially mathematically-tricky code. Spend the extra effort on something interesting instead.

Overhead. In a numerically-intensive program, the overhead of calling a few function is absolutely tiny relative to the computational work, and the overhead of "not needing to communicate" is virtually nil. I would not worry about a few function calls. It would matter in a non-optimized tight inner loop, but for API calls this is not a consideration.

MPI. I don't know how to easily write a program in a way that exactly the same program would easily compile and work both with MPI and without any MPI at all. It is probably far too much effort to try and hide all the MPI calls with ifdefs.

One way is to have two programs, MPI and non-MPI, that rely on the same core mathematical routines. Another way is to make sure the serial version is identical to the MPI version running in a single process. The only overhead here you need to take note of is the overhead of setting up the MPI environment: you need to make sure it's not included in any timing results for the computation, MPI or non-MPI. The overhead of "communicating $0$ bytes" should be nil.

OpenMP. Given how much of OpenMP's parallelism is accessed through compiler pragmas, rather than explicit API calls, many OpenMP programs will compile with and without OpenMP support. It is also probably not worth the effort to hide OpenMP API calls with ifdefs: just run the program with only one thread.

• Some good advice here, especially related to minimizing duplicate code. I also agree that it is impractical (or at least not worth the effort) to hide MPI/openMP calls. However, testing every possible path through your code is almost always impractical. Rather, each component of the code should be tested independently. This is the advantage of writing modular or object-oriented code. If the components all work independently (with well defined in/outputs and no side effects) then they can be combined with confidence regardless of conditional statements. – Doug Lipinski Jun 21 '14 at 21:13
• @DougLipinski You're right and I agree with you, except for placing emphasis. The interaction between those components is what the program's behaviour is made of. So using modular separately-tested components solves the problem of having to test absolutely everything, but the interactions are still there and need to be tested fully. I'd rather advise someone to test everything fully, and maybe they'll use modular design as a technique for making it easier. – Kirill Jun 21 '14 at 21:47
• Testing isn't an either-or decision between unit tests and integration tests. It is common in serial codes to employ both unit tests and integration tests. The lack of an MPI-capable unit testing framework makes this process more labor-intensive for distributed memory parallel codes, but not impossible. I wouldn't advise integration tests without unit tests, because unit tests are more diagnostic than integration tests; by construction, if you test interactions without testing individual components, there are more potential points of failure to examine. – Geoff Oxberry Jun 23 '14 at 16:26
• @GeoffOxberry I'm definitely not saying one should prefer integration tests to unit tests or vice versa. – Kirill Jun 24 '14 at 3:09

Something in your description sounds amiss when you say you need four codes. You may want clarification from your instructor.

Would one MPI+OpenMP code that produces correct results for any "OMP_NUM_THREADS=n mpiexec -np m" invocation satisfy your assignment? That would permit serial, parallel, multithreaded, and hybrid operation from a single binary at runtime.

• Upon seeing the question, a single code approach was my first thought as well. – Geoff Oxberry Jun 23 '14 at 16:16
• I suppose I will be going with single code and simply setting flags to 1 when the code is meant to be executed serially. Thank you for your help. – kfkhalili Jun 24 '14 at 11:48
• You're welcome. Generally, people try to write a single Hybrid code for such things and then be careful that it degenerates correctly in the MPI-only, Multicore-only, and Serial-only cases. Makes debugging much easier to work mostly with a serial code and then to verify one gets the right results when running more exotically. – Rhys Ulerich Jun 24 '14 at 14:56

Let's say for simplicity that you have a 1D problem, and you discretize it over N grid points. Your code will have a bunch of loops from 1 to N that do stencil operations -- assuming you're not doing implicit time stepping because solving linear systems directly complicates the story considerable.

With this setup, a serial, OpenMP, and MPI version look remarkably similar.

• Your serial version will have the loops from 1 to N.
• Your OpenMP version has the exact same loops, except with a directive that they are parallel.
• Interestingly, the MPI version will still have the same loops, but now your 1..N space is divided over your processors

Ok? So the bulk of your code will look the same in all three cases. Now for the MPI case, which is the hard one. Your big loop gets divided over the processes, so you'd think that one process gets indices 100-200 for instance. Yes, but only in a metaphoric sense. With MPI every process is independent, so it thinks it has addresses 0-100, and you need to translate this back to the global numbering as needed.

The real problem with MPI is that when you apply a stencil you may need some array components that live on another process. Elements 99 and 201 in the above example. That's where you need to do some send and receive operations before you execute your stencil code.

So you have only one code. Can you compile it three ways? Yeah, just sprinkle some #ifdef directives through it. That part is simple.