How do I get reliable timing data for time spent in function calls in my code?

Question

This question is a follow-up to Fortran: Best way to time sections of your code?.

If I want to time functions in my code, I know I could use gprof or kcachegrind. I also know that the results from these tools can be skewed (see http://www.yosefk.com/blog/how-profilers-lie-the-cases-of-gprof-and-kcachegrind.html and https://stackoverflow.com/questions/1777556/alternatives-to-gprof/1779343#1779343).

I know I could add manual timers to each function for which I want data, which can be tedious or impractical for libraries, if I want data for everything.

Unfortunately, I run into communities that want this timing data to use as evidence in arguing for the performance their methods (to demonstrate improvement in performance, point out spots where performance is bad, for scientific papers, and so on). This seems to be popular with management-types and some academic-types. Is there a better way to get reliably accurate timing data than inserting timers? Should I be using a combination of imperfect tools and sifting through the performance data in some way?

(Note: This question isn't about performance tuning, even though it's related. You can do performance tuning without timing things by using random pausing. It also isn't about whether or not timing is worthwhile, because these communities want timing data, and I don't have the power to change their mind easily. Any comments about these topics are great discussion, but they're not helpful in answering my question, because the reality is that the people I answer to want timing data that somehow reflects performance.)

Answer 1

You might consider a stack sampling profiler like HPCToolkit or VTune, or the system profiler for Linux, prof.

Also, I don't see what's objectionable about wanting to know how long things take. If you want to demonstrate that your implementation of an algorithm has the asymptotic performance you derived, actually measuring the running time is the best way to do so.

Answer 2

While it may be true that gprof or valgrind's cachegrind can produce skewed results, they are almost always good enough for what you really want to do -- namely find out which functions are "expensive" and which are "not expensive". As the article you quote shows, it's possible to generate programs for which profilers do not show the whole story, or in fact even show falsehoods. However, when you apply profiles to "real programs", they do almost always give you a fairly good picture of the truth, and that is precisely why they are still used.

In other words, despite their limitations, I do believe that when applied to real-world programs, profiles do show very useful data and I would not hesitate to include this data into publications (with a short description of said limitations).

Answer 3

I've often used Intel's VTune Amplifier to get precise timings, which, on the right hardware, will break timings down to the instruction level. The better results come from using on-chip counters, i.e. the Performance Monitoring Unit.

The counts are still not exact, but have a much better resolution than what you can get out of software-based collectors.

As for your comment on the need for exact numbers, I don't completely agree. I don't consider myself to be a bean counter, but I rely heavily on timer, mostly implemented directly in the code using calls to cycle.h from FFTW wrapped in macros, for actual research. Specifically, I work on algorithms for task-based parallelism, and need good estimates of the time spent doing actual work vs. overheads in the task allocation. These overheads are usually the sum of many small function calls and can be difficult to assess in a profiler, but are pretty much the only measure of how good a scheduling scheme is. In this case, precise timers are actually a necessary condition to doing good research.