I am comparing the performance several finite difference methods of solving an initial-boundary value problem. There are several dimensions to this comparison:
- Number of cells
- Number of timesteps
- Solution Method:
- Explicit (one sweep on a single thread, no iterations)
- Alternating Direction Explicit (two sweeps on separate threads, no iterations)
- Alternating Direction Implicit (two sweeps on one thread [TDMA] with three sub-timesteps, no iterations)
- Fully Implicit (any number of threads, using an iterative BICGSTAB solver)
My question is this: What metric should I use to compare the relative performance for various combinations of number of cells, number of timesteps, and solution method?
There is some information on performance metrics on David J. Lilja's website. He seems to consider execution time (wall and CPU) as the best metric. But I'm wondering if there might be a more suitable metric for my application.
Here is what I've considered so far:
- Wall time and CPU time: This is easy to measure, but the problem is that it is only representative for computer architecture where the program is executed. I'm doing most of my testing on a 24 core machine...which won't be representative of typical applications. It would be nice not to have to include a description of the machine's architecture every time I publish results. Are there meaningful ways to normalize measured time to avoid this problem?
Number of Operations:
Sum of (N_iterations*N_sweeps*N_sub-timesteps*N_cells)/(N_threads) across all timesteps.
This is repeatable and platform independent, but it doesn't include some of the overhead of setting up the initial-boundary value problem and creating matrices, etc. It also makes some simplifying assumptions about threading overhead.
It would also be nice to give a rough estimate of parallel and serial performance, so that someone could get a sense of what the speed up would be for a given number of processors (i.e., a way to extrapolate from the results on my 24 core machine).