# Why are "instructions retired" more stable than "cycles" or "task-clock" when using "perf"?

I tested the perf tool in Ubuntu 18.04 on a simple benchmark in our compiler (parsing some file). I ran perf several times and put here the slowest:

$sudo perf stat ./parse Construct Parse Parsing: 93ms Counting: 14ms Total: 107ms Count: 450009 String size (bytes): 2250042 Allocator usage (bytes): 48400928 Performance counter stats for './parse': 112.329506 task-clock (msec) # 0.999 CPUs utilized 0 context-switches # 0.000 K/sec 0 cpu-migrations # 0.000 K/sec 14,560 page-faults # 0.130 M/sec 418,991,316 cycles # 3.730 GHz 843,385,199 instructions # 2.01 insn per cycle 139,033,655 branches # 1237.730 M/sec 1,033,015 branch-misses # 0.74% of all branches 0.112438670 seconds time elapsed  and fastest run: $ sudo perf stat ./parse
Construct
Parse
Parsing: 80ms
Counting: 12ms
Total: 93ms
Count: 450009
String size (bytes):      2250042
Allocator usage (bytes): 48400928

Performance counter stats for './parse':

97.922823      task-clock (msec)         #    0.998 CPUs utilized
0      context-switches          #    0.000 K/sec
0      cpu-migrations            #    0.000 K/sec
14,559      page-faults               #    0.149 M/sec
376,416,206      cycles                    #    3.844 GHz
843,259,218      instructions              #    2.24  insn per cycle
138,973,382      branches                  # 1419.213 M/sec
1,031,451      branch-misses             #    0.74% of all branches

0.098073332 seconds time elapsed


When comparing the various statistics, here are the percentage improvements from the slowest to the fastest run:

task-clock      12.83% faster
cycles          10.16% less
frequency        3.06% faster
instructions     0.015% less
insn per cycle  11.44% more
branches         0.043% less
branch-misses    0.15% less


Why is instructions 3 orders of magnitude more stable than the total time (task-clock) or cycles?

One explanation could be that my CPU (Intel(R) Xeon(R) CPU E3-1505M v6 @ 3.00GHz) executes roughly the same number of instructions each time, but depending on various factors (temperature, past history, other programs running in linux at the same time, etc.) it can sometimes execute ~10% more instructions per cycle by hiding latency / executing instructions in parallel, which means it needs ~10% less cycles to execute all of them, and given the fact that the frequency is roughly the same (within ~3%) this translates to ~10% less time. Is this a correct explanation?

• This could be wrong but maybe the layout of data in heap memory isn't the same from one invocation to the next? Then the variance comes from number of cache misses. I wonder how this changes if for example you link against jemalloc or tcmalloc instead? May 14 '20 at 19:57
• I mean, I'm surprised that the number of instructions isn't identical from run to run. May 14 '20 at 22:26