# Profiling CFD code with Callgrind

I'm using Valgrind + Callgrind to profile a solver I have written. As the Valgrind user manual states, I've compiled my code with the debugging options for the compiler:

"Without debugging info, the best Valgrind tools will be able to do is guess which function a particular piece of code belongs to, which makes both error messages and profiling output nearly useless. With -g, you'll get messages which point directly to the relevant source code lines."

Valgrind manual

When compiled with the debugging option, the codes run much slower. CFD code, becomes REALLY slow, even for small cases when compiled with debugging flags. Valgrind makes it 40x slower (see the manual 1).

1. What tools are you using for code profiling (profiling, not benchmarking)?

2. How long do you let the code run (statistics: how many time steps)?

3. How large are the cases (if the case fits in the cache, the solver is orders of magnitude faster, but then I'll miss the memory related processes)?

• You can compile the code with both debugging symbols and optimization enabled. Still, 40x through valgrind (which simulates all memory access) is not unreasonable. – Aron Ahmadia Feb 27 '12 at 10:44
• Thanks, this is what I've read as well... what I would like to know are infos about the every-day experiences in profiling (with valgrind preferably): how much time is normal to wait for the reports, how many iteration counts do I need, what can I exclude... etc... – tmaric Feb 27 '12 at 11:02
• Your question is also a bit broad. I recommend editing your question down to focus on Q2.1 and Q2.2, as Q1 is a completely different question (I'm happy for you to ask it separately, it's a good one, but phrase it as "Which tools would you use to solve problem X", where X is well-described!), whereas Q2 on its own is too general. – Aron Ahmadia Feb 27 '12 at 12:21
• Can you also edit to name callgrind, cachegrind, or massif. Many people associate Valgrind only with the default tool (memcheck). As an emulation-based (rather than interrupt-based) profiling system, you should not need to run for a long time. – Jed Brown Feb 27 '12 at 12:56
• @Aron & Jed: thanks for the tips, I've edited the question. :) – tmaric Feb 27 '12 at 13:01

Q1: What tools are you using for code profiling (profiling, not benchmarking)?

Q2: How long do you let the code run (statistics: how many time steps)?

Q3: How large are the cases (if the case fits in the cache, the solver is orders of magnitude faster, but then I'll miss the memory related processes) ?

Here's an example of how I do it.

I separate the benchmarking (seeing how long it takes) from the profiling (identifying how to make it faster). It is not important that the profiler be fast. It is important that it tell you what to fix.

I don't even like the word "profiling" because it conjures an image something like a histogram, where there's a cost-bar for each routine, or "bottleneck" because it implies there's just a little place in the code that needs to be fixed. Both of these things imply some sort of timing and statistics, for which you assume accuracy is important. It is not worth giving up insight for accuracy of timing.

The method I use is random pausing, and there's a full case study and slide show here. Part of the profiler-bottleneck world view is that if you don't find anything, there's nothing to be found, and if you do find something and get a certain percent speedup, you declare victory and quit. Profiler fans almost never say how much speedup they get, and the ads only show artificially contrived problems designed to be easy to find. Random pausing finds the problems whether they are easy or hard. Then fixing one problem exposes others, so the process can be repeated, to get compounded speedup.

In my experience from numerous examples, here's how it goes: I can find one problem (by random pausing) and fix it, getting a speedup of some percent, say 30% or 1.3x. Then I can do it again, find another problem and fix it, getting another speedup, maybe less than 30%, maybe more. Then I can do it again, multiple times until I truly can't find anything else to fix. The ultimate speedup factor is the running product of the individual factors, and it can be amazingly large - orders of magnitude in some cases.

INSERTED: Just to illustrate this last point. There's a detailed example here, with slide show and all files, showing how a speedup of 730x was achieved in a series of problem removals. The first version took 2700 microseconds per unit of work. Problem A was removed, bringing the time down to 1800, and magnifying the percentages of remaining problems by 1.5x (2700/1800). Then B was removed. This process continued through six iterations, resulting in almost 3 orders of magnitude speedup. But the profiling technique has to be really efficacious, because if any of those problems are not found, i.e. if you reach a point where you incorrectly think nothing more can be done, the process stalls.

INSERTED: To put it another way, here's a graph of the total speedup factor as successive problems are removed:

So for Q1, for benchmarking a simple timer suffices. For "profiling" I use random pausing.

Q2: I give it enough workload (or just put a loop around it) so it runs long enough to pause.

Q3: By all means, give it realistically large workload so you don't miss cache problems. Those will show up as samples in the code doing the memory fetches.

• Mike, do you have a preference for how to do random pausing in the absence of a visual IDE? Can this process be automated in some way? – Matthew Emmett Jun 14 '12 at 19:43
• @Matthew: I understand there are tools like pstack and lsstack, but I really consider this a process more in common with debugging. So even if the best debugger I can bring to bear is gdb, it gets the job done. With a debugger you can examine data, and that can make the difference when the stack alone doesn't tell you enough. – Mike Dunlavey Jun 14 '12 at 20:52

The poor man's profiler is basically a gdb script that samples the call stack. You'll still need to have debugging symbols. It is still slow, but as it does not implement a virtual machine to run the code on it is often faster than callgrind and adequate to the task.

I've run in on particle physics analyzers with modest success (i.e. I demonstrated that the code didn't any horrible hot spots and optimization was going to require a better algorithm).

• + Absence of evidence is not evidence of absence :) What that poor man's profiler should do is take fewer traces and not collapse them, but let you see them. The human eye is far better at detecting useful patterns than simple function time estimates, and if you see something you could improve on as few as 2 samples, it will help significantly. The fraction X it will save is a beta distribution with mode 2/N, where N is how many traces you examined, and the speedup factor will be 1/(1-X), which can be large. – Mike Dunlavey Apr 4 '12 at 13:58

To add to the available great answers, there is a tool developed at Rice which automates stack sampling and therefore has very little overhead:

http://hpctoolkit.org/

• That looks nice, though (sorry) I've put on my flame hat here. I don't tune on compiler-optimized code because it's hard to see what's going on in mangled code. The things I'm pruning are not things the optimizer could deal with - like calling exp and log with same arguments repeatedly, or matrix operations spending all their time decoding options. I tune as far as I can, then turn on -O3. – Mike Dunlavey Sep 10 '13 at 13:04
• Tools are tools, and are only useful if the user knows and understands their limitations. I don't think there will ever be a "perfect profiler" which will remove the user from the equation altogether with respect to understanding its output and knowing how to use the information. – Reid.Atcheson Sep 10 '13 at 16:04

Allinea MAP is a commercially developed and supported sampling profiler and hence - like HPC Toolkit suggested in an earlier answer - can run on production sized jobs if you wish.

This kind of tool points to CPU bottlenecks or poor MPI communication, but also the whole oversight of profiling the entire job can be priceless in finding surprise issues.

There are often low-hanging performance fruits to be had that are outside of the core kernel of a CFD code, in areas that were not expected. Randomized stack sampling is - whether done manually with GDB, or with tools like HPC Toolkit and Allinea MAP - the best way to find them. If something is important to performance it will show up.