I currently use FEniCS and Deal.II to solve various FEM problems. I am also writing my own implementation of these problems by directly implementing the data structures, routines, and solvers within PETSc. What kind of comparisons can I draw between these three various implementations given the same FEM problem/discretization? Particularly in the context of high performance and parallel computing.
Naturally, the first thing to do is check the numerical accuracy - do I get the correct solutions. Then I could check the strong scaling, weak scaling, and associated wall-clock time across multiple processing cores. But what else can I do? For instance, is there a way to determine how "efficient" these frameworks are algorithmically and in terms of machine hardware?
My goal is to show people which of these implementations will yield the fastest possible time given a specific problem size and FE discretization, but I think it would also be interesting to show how "credible" these timings are. For example:
1) Why does implementation X take 45 seconds to solve a problem but implementations Y and Z only take 30 and 25 seconds respectively?
2) Why can't any of these three take only 4.5 seconds?
3) How much of the time within the computational framework is spent doing useful work versus waiting on/accessing memory/cache?
Sure memory bandwidth is likely the answer to why these implementations don't achieve the ideal or perfect time, and benchmarks like STREAM can tell you what you can expect given your machine. I am wondering if anyone has a methodology, tool, or suggestion to properly determine or quantify these.