Profiling numeric libraries

Question

What efforts have been made to quantify the costs of the common numeric workhorse operations such as those found in LAPACK, BLAS, sparse matrix operations, etc....

We have several libraries which implement identical interfaces found in BLAS and LAPACK. I suspect these all have different strengths and weaknesses on various architectures.

Has there been any effort to compare them as a group for a particular sequential architecture?

Has there been any effort to do this on more complex architectures, i.e. has anyone compared the various GPU or multi-core library implementations?

In general each library has it's own set of papers comparing a few operations with another (often less sophisticated) library. I'm looking for work which does a more substantial and broad review.

In a perfect world we would like to be able to as questions like "for an i7 chip with 4GB memory and roughly this memory hierarchy the LU decomposition is 20% faster using library X over library Y". Profiling every algorithm on every architecture is obviously infeasible. What work is feasible? Do we know generally when library X is preferable to library Y without understanding each library in depth?

Answer 1

Not only has there been effort, there is millions of dollars a year poured into the benchmarking by industry every year to prove their machine is better.

The HPC Challenge Benchmarks are a set of programs that can be reimplemented by and submitted to the HPCC board. They give out reports on each architecture each year.

Now comparing various library implementations is often a challenge, and probably something that library developers don't really want. If you design a library to do X and another library beats you at Y, then why do you care. As a user I'm quite skeptical of such comparisons as well. They usually miss key features of a library.