# What software is good to use for parallel debugging?

I'm not running any parallel code right now, but I anticipate running parallel code in the future using a hybrid of OpenMP and MPI. Debuggers have been invaluable tools for me when running serial projects.

Can anyone recommend a parallel debugger (or multiple debuggers) to use for debugging parallel software? Free software would be preferable, but don't hesitate to mention effective commercial software.

• I don't see how the answers here will differ significantly from stackoverflow.com/questions/329259/…. MPI is the hard part here, not the OpenMP. In any case, debugging race conditions in threaded programs is borderline unsolvable right now. – Jeff Hammond Jul 23 '13 at 6:44
• ThreadSanitizer is a good solution for debugging race conditions in threaded programs, though I know of nobody who's tried to add MPI to the mix! – mabraham Aug 26 '15 at 21:06

There are basically two major, commercial choices out there: DDT from Allinea (which is what we use at TACC) and Totalview (as mentioned in the other comment). They have comparable features, are both actively developed, and are direct competitors.

Eclipse has their Parallel Tools Platform, which should include MPI and OpenMP programming support and a parallel debugger.

• I've never heard of anyone using the PTP parallel debugger. I'm not sure what that means... – Jeff Hammond Jul 23 '13 at 6:50
• I have a few colleagues that have tried, but I've never played with it myself. – Bill Barth Jul 23 '13 at 19:31
• For parallel programs on the GPU, there's also CUDA-GDB. – Anderson Green Dec 17 '20 at 3:48

I must give the curmudgeon answer. My productivity has never been improved by any of the suggestions above. They are slow and expensive compared to my preferred option in parallel: one gdb session per process. Each gdb can connect to an MPI process and sit in an xterm (this happens automatically in PETSc using -start_in_debugger). I have used this for 15 years, happily. Objections:

1) I can't look at global data

Since MPI is a shared-none model, there is no global data, only local data

2) This strategy does not scale to lots of processes

Neither do bugs. Bugs happen on individual processes, maybe with input from 1 or 2 neighbors. YOu can easily spawn gdb only on the participating processes (in PETSc you use -debugger_nodes 0,5,17 for example). Also, the above systems give up a lot when the run on every process, which makes them slow. The gdb method is, in fact, much more scalable.

gdb is also very portable. It runs everywhere, understands C++ and Fortran, and allows you to execute arbitrary code inside the run. I have written special functions to easily display data when running in it.

• Hey Coward, if you downvote, leave a comment. – Matt Knepley Dec 14 '11 at 16:48
• I wasn't the down vote, but I do disagree to some extent. I've encountered a few bugs at scale that didn't show at small sizes, and using a parallel debugger was an efficient way to find them. I do most of my debugging with printf and attaching to individual processes with gdb, but I have seen the benefit of having a parallel debugger. – Bill Barth Dec 14 '11 at 17:08
• The only time I have ever encountered a bug at scale was a performance bug due to an improper collective communication algorithm being chosen. Then again, my view is even more extreme than Matt's, as the closest thing to a debugger that I ever use is valgrind. – Jack Poulson Dec 14 '11 at 17:19
• @BillBarth I know you are right that bugs exist on 1000 proces that do not show up on smaller problems (Dinesh had a famous PETSc one that lasted for months that only showed up on 82 procs). My point was more to counter the prevailing wisdom. I think parallel debuggers are a good last resort, not first resort. – Matt Knepley Dec 14 '11 at 17:20
• I downvoted you. Your answer not what was asked. – aterrel Dec 14 '11 at 21:08

I use only two debuggers for serial and parallel programs:

1. The Kernighan debugger, i.e. judicious print statements and careful thinking.
2. Multiple instances of GDB as described o http://www.open-mpi.org/faq/?category=debugging#serial-debuggers.

In the case where (2) is not sufficiently scalable, I refer to (1b).

• I've never heard the name "Kernighan debugger", but I approve, as it is how I always debug. – Jack Poulson Jul 23 '13 at 7:26

There is Intel Parallel Studio which includes a parallel debugger. I've never worked with it but I've seen it used in a few demos. Here's a video tutorial that shows some of the features.

I've also seen a few wrappers around gdb that worked reasonably well in certain cases.

Totalview. It's a commercial debugger. It's very easy to view the stack on each processor. You can see variable values (and change them) across processors/threads. You can plot vectors or matracies to visualize variable values. Apparently scripting is possible too (Tk/Tcl), for sofisticated watch point analysis, though I've never worked with this myself.

• On the subjective side, when my university's HPC center installed this I thought it was overkill. Then I found out how easy it was to do very complicated debugging. It's really a great program. – Yann Dec 14 '11 at 3:30
• I second totalview too. I have used it in many instances and it is extremely powerful, albeit highly expensive... – BlaB May 30 '17 at 17:10

For a couple simple ways to debug parallel codes, we've collected a few answers in the deal.II FAQs in the section on debugging: https://github.com/dealii/dealii/wiki/Frequently-Asked-Questions#debugging-dealii-applications

I wonder why no one mentioned Padb (Parallel Application Debugger) which is open source and free software as the OP prefers, but not as powerful as commercial counterparts for example: TotalView for HPC

Here's a digest of some answers given to me previously :

OpenMP has timing functions: omp_get_wtime() and omp_get_wtick() -- online docs