I must give the curmudgeon answer. My productivity has never been improved by any of the suggestions above. They are slow and expensive compared to my preferred option in parallel: one gdb session per process. Each gdb can connect to an MPI process and sit in an xterm (this happens automatically in PETSc using
-start_in_debugger). I have used this for 15 years, happily. Objections:
1) I can't look at global data
Since MPI is a shared-none model, there is no global data, only local data
2) This strategy does not scale to lots of processes
Neither do bugs. Bugs happen on individual processes, maybe with input from 1 or 2 neighbors. YOu can easily spawn gdb only on the participating processes (in PETSc you use
-debugger_nodes 0,5,17 for example). Also, the above systems give up a lot when the run on every process, which makes them slow. The gdb method is, in fact, much more scalable.
gdb is also very portable. It runs everywhere, understands C++ and Fortran, and allows you to execute arbitrary code inside the run. I have written special functions to easily display data when running in it.