27
$\begingroup$

I'm not running any parallel code right now, but I anticipate running parallel code in the future using a hybrid of OpenMP and MPI. Debuggers have been invaluable tools for me when running serial projects.

Can anyone recommend a parallel debugger (or multiple debuggers) to use for debugging parallel software? Free software would be preferable, but don't hesitate to mention effective commercial software.

$\endgroup$
2
  • $\begingroup$ I don't see how the answers here will differ significantly from stackoverflow.com/questions/329259/…. MPI is the hard part here, not the OpenMP. In any case, debugging race conditions in threaded programs is borderline unsolvable right now. $\endgroup$ Jul 23, 2013 at 6:44
  • $\begingroup$ ThreadSanitizer is a good solution for debugging race conditions in threaded programs, though I know of nobody who's tried to add MPI to the mix! $\endgroup$
    – mabraham
    Aug 26, 2015 at 21:06

8 Answers 8

18
$\begingroup$

There are basically two major, commercial choices out there: DDT from Allinea (which is what we use at TACC) and Totalview (as mentioned in the other comment). They have comparable features, are both actively developed, and are direct competitors.

Eclipse has their Parallel Tools Platform, which should include MPI and OpenMP programming support and a parallel debugger.

$\endgroup$
3
  • $\begingroup$ I've never heard of anyone using the PTP parallel debugger. I'm not sure what that means... $\endgroup$ Jul 23, 2013 at 6:50
  • $\begingroup$ I have a few colleagues that have tried, but I've never played with it myself. $\endgroup$
    – Bill Barth
    Jul 23, 2013 at 19:31
  • $\begingroup$ For parallel programs on the GPU, there's also CUDA-GDB. $\endgroup$ Dec 17, 2020 at 3:48
17
$\begingroup$

I must give the curmudgeon answer. My productivity has never been improved by any of the suggestions above. They are slow and expensive compared to my preferred option in parallel: one gdb session per process. Each gdb can connect to an MPI process and sit in an xterm (this happens automatically in PETSc using -start_in_debugger). I have used this for 15 years, happily. Objections:

1) I can't look at global data

Since MPI is a shared-none model, there is no global data, only local data

2) This strategy does not scale to lots of processes

Neither do bugs. Bugs happen on individual processes, maybe with input from 1 or 2 neighbors. YOu can easily spawn gdb only on the participating processes (in PETSc you use -debugger_nodes 0,5,17 for example). Also, the above systems give up a lot when the run on every process, which makes them slow. The gdb method is, in fact, much more scalable.

gdb is also very portable. It runs everywhere, understands C++ and Fortran, and allows you to execute arbitrary code inside the run. I have written special functions to easily display data when running in it.

$\endgroup$
9
  • 4
    $\begingroup$ Hey Coward, if you downvote, leave a comment. $\endgroup$ Dec 14, 2011 at 16:48
  • 5
    $\begingroup$ I wasn't the down vote, but I do disagree to some extent. I've encountered a few bugs at scale that didn't show at small sizes, and using a parallel debugger was an efficient way to find them. I do most of my debugging with printf and attaching to individual processes with gdb, but I have seen the benefit of having a parallel debugger. $\endgroup$
    – Bill Barth
    Dec 14, 2011 at 17:08
  • 3
    $\begingroup$ The only time I have ever encountered a bug at scale was a performance bug due to an improper collective communication algorithm being chosen. Then again, my view is even more extreme than Matt's, as the closest thing to a debugger that I ever use is valgrind. $\endgroup$ Dec 14, 2011 at 17:19
  • 1
    $\begingroup$ @BillBarth I know you are right that bugs exist on 1000 proces that do not show up on smaller problems (Dinesh had a famous PETSc one that lasted for months that only showed up on 82 procs). My point was more to counter the prevailing wisdom. I think parallel debuggers are a good last resort, not first resort. $\endgroup$ Dec 14, 2011 at 17:20
  • 3
    $\begingroup$ I downvoted you. Your answer not what was asked. $\endgroup$
    – aterrel
    Dec 14, 2011 at 21:08
7
$\begingroup$

I use only two debuggers for serial and parallel programs:

  1. The Kernighan debugger, i.e. judicious print statements and careful thinking.
  2. Multiple instances of GDB as described o http://www.open-mpi.org/faq/?category=debugging#serial-debuggers.

In the case where (2) is not sufficiently scalable, I refer to (1b).

$\endgroup$
1
  • 1
    $\begingroup$ I've never heard the name "Kernighan debugger", but I approve, as it is how I always debug. $\endgroup$ Jul 23, 2013 at 7:26
4
$\begingroup$

There is Intel Parallel Studio which includes a parallel debugger. I've never worked with it but I've seen it used in a few demos. Here's a video tutorial that shows some of the features.

I've also seen a few wrappers around gdb that worked reasonably well in certain cases.

$\endgroup$
3
$\begingroup$

Totalview. It's a commercial debugger. It's very easy to view the stack on each processor. You can see variable values (and change them) across processors/threads. You can plot vectors or matracies to visualize variable values. Apparently scripting is possible too (Tk/Tcl), for sofisticated watch point analysis, though I've never worked with this myself.

$\endgroup$
2
  • $\begingroup$ On the subjective side, when my university's HPC center installed this I thought it was overkill. Then I found out how easy it was to do very complicated debugging. It's really a great program. $\endgroup$
    – Yann
    Dec 14, 2011 at 3:30
  • $\begingroup$ I second totalview too. I have used it in many instances and it is extremely powerful, albeit highly expensive... $\endgroup$
    – BlaB
    May 30, 2017 at 17:10
1
$\begingroup$

For a couple simple ways to debug parallel codes, we've collected a few answers in the deal.II FAQs in the section on debugging: https://github.com/dealii/dealii/wiki/Frequently-Asked-Questions#debugging-dealii-applications

$\endgroup$
1
$\begingroup$

I wonder why no one mentioned Padb (Parallel Application Debugger) which is open source and free software as the OP prefers, but not as powerful as commercial counterparts for example: TotalView for HPC

$\endgroup$
-1
$\begingroup$

Here's a digest of some answers given to me previously :

OpenMP has timing functions: omp_get_wtime() and omp_get_wtick() -- online docs

Google has a CPU profiler

There's Scalasca that does OpenMP and MPI profile and analysis

Then there's Tau and vtune which I haven't used.

Good luck!

$\endgroup$
3
  • $\begingroup$ I don't think the question is about timing, but I might be wrong. Good suggestions though... $\endgroup$
    – Yann
    Dec 14, 2011 at 15:44
  • $\begingroup$ This answer is more about profiling than debugging... $\endgroup$
    – mbq
    Dec 14, 2011 at 15:44
  • $\begingroup$ I have found that profiling tools make good substitutes for parallel debuggers. I find it often the case that parallel bugs are related to performance problems, such as logjam in MPI. Performance tools will often reveal this. TAU's memory profiler is good for figuring out why random segfaults may occur. $\endgroup$ Jul 23, 2012 at 2:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.