Of the criteria you've specified, I think the closest project that I know of would be the University of Florida sparse matrix collection. People routinely use this data set to compare sparse linear algebra solvers, and you can filter by application, number of nonzeros, dimensions of matrix, and so on with a really nice web interface, MATLAB interface, or Java GUI. I've seen tables of these problems listed in papers along with solver run time comparisons to 4 to 8 linear algebra solvers.
I agree that it would be useful to compile such databases, and furthermore, I think the UF sparse matrix collection approach for compiling the data is an excellent one, and would make a great start for anyone thinking about realizing this idea. Running all of the problems, in practice, doesn't seem like a major difficulty as long as you can get access to all of the solvers; if you have access to the solvers, and a reliable standard reference machine with all of the necessary software installed, then it should be a matter of running a script and collecting the data. The difficulty, in my mind, would be getting people to give you their software, if it isn't open source. If it's commercial, you could buy it, or even possibly get people to donate the software, which is something I know that happens in the optimization community when people are building portable software interfaces for commercial codes, such as in the COIN-OR project. But if it's research software that is neither commercial nor open source, then you need to convince people to buy into the endeavor, and they may not trust a third party to assess their software fairly.
I also know that in optimization, there are downloadable databases of problems (CUTEr
comes to mind) and books of test problems for optimization. I have seen people (for instance, I'm specifically thinking of a talk by Ruth Misener at AIChE 2011) compare their optimization solver versus other solvers on databases of problems in presentations; I'm not sure what gets released publicly. I know that there's a tradition in optimization for comparison on a large scale (many solvers, many problems); I just don't think there's an online database available.
Another thing that I think is important is that we distinguish here between methods and software implementations. In scientific computing, we all talk about which methods are faster or slower based on things like computational complexity metrics, or our experiences with various problems. When it comes to measuring computational time quantitatively, however, unless one counts the number of FLOPs in a particular algorithm, one has to implement the algorithm in software and then measure the performance in some way (memory usage, wall clock time of execution, etc.). It makes sense to assess the performance of a method when looking at computational complexity or FLOP counts, because we don't need an implementation to measure such things, but the moment we're interested in actual wall clock run times, talking about methods is only useful as an abstract, colloquial device. (For instance, sparse direct solvers are fast on sufficiently small sparse problems with good fill-reducing heuristics, such as nested dissection.)
I bring up this distinction between methods and software because in such a database, I could also see the possibility of tracking the improvement in software over time. So, for instance, with something like, say, PETSc, or PyCLAW, or whatever software is being tested, it would be interesting to see what problems are affected positively (or negatively!) by upgrades in the software. This could be useful for researchers trying to decide if it's worth any potential costs in money and manpower to upgrade their codes. Another reason such a distinction is important is because a good method can be implemented badly; I think this possibility contributes to the reticence people sometimes have in sharing their research codes. A database of implementations would be able to distinguish between good and bad implementations of the same method (or types of methods).
I think whatever comes of this idea (and I hope something comes of it, and would be willing to contribute after my PhD), it's important to emphasize that distinction between software and methods, because if we're running test problems, we're going to be posting results for software.