Performance based on the Roofline model

Question

To somewhat follow up on the question asked here, I have been told that the Roofline model is one way of assessing the performance of any scientific code. Basically I compute the Arithmetic Intensity (ratio of FLOPS over DRAM bytes) and multiply this by the STREAM bandwidth to obtain the ideal FLOPS/s. I could also use the AI to see how close I get to the maximum performance of my given machine.

That said, what is the easiest way to obtain this AI for finite element packages like FEniCS or Deal.II? To simply things for now, I am not too concerned with register/cache reuse or quantification of useful bandwidth sustained for some level of cache. If I wrote my own explicit finite element implementation using PETSc, I could simply count by hand the approximate number of FLOPS and approximate the number of load/stores from all vector operations and sparse matrix-vector multiply as outlined here. However, does anyone have any suggestions for doing this for any given implementation?

Answer 1

What you absolutely cannot do is measure the flops on any modern Intel architecture using the hardware performance counters. Starting with Nehalem, Intel chips essentially count floating point instructions issued, and if the data hasn't arrived to the processor registers, the instruction may not get retired. Instead it will get reissued some time later, and this keeps happening until the data is available and the instruction can finish. As a result, hardware PMC measurements of STREAM Triad over count flops by a factor of 10. DGEMM is over counted by a few percent. These probably represent the bounds. This got so bad that Intel disabled the flop performance counters on Haswell.

As such, your only real option is to open the code and manually count the operations unless some of these libraries include instrumentation that counts them for you.

In some sense, why go to this trouble? Why not do two things, since you're going to be setting up and running these codes anyway:

1. Run exactly the same problem in each code on a given architecture compiled the same way to the best of your ability
2. Run each case for different values of the size parameter (grid points, elements, whatever)

With the run times of each of these cases, you should be able to fit a polynomial or other complexity theory function to run time data thereby confirming that the method has the expected computational complexity and also estimating the various associated constants. I.e. if you expect $O(n^2)$, then you should get a good fit with a quadratic. You may need to ignore the small end of $n$ entirely. You should also be able to see which one is faster. Unless these curves cross multiple times (which they really shouldn't once you get into the asymptotic regime), the one that is fastest is the most efficient. Make sure you run large enough.

Asking whether any one of these codes is as efficient as it theoretically could be is a much harder question.