Databases of results for numerical codes

Question

In the numerical methods literature, many research papers consist of a description of a new algorithmic variation, followed by a few test problems comparing the new method with one or two existing methods. This makes it difficult to determine

• How the new method would perform on other problems of interest
• How the new method would compare to yet other existing methods

Of course, both of these questions are usually critical for someone deciding whether to adopt the new method. In order to improve this situation, it seems desirable to, if possible, get people to run a large set of test problems (see this question) with their method and catalog the results in a database for comparison with other methods. A scientist or engineer could then query the database, perhaps indicating what solution/problem characteristics are most important to them (accuracy, efficiency, qualitative properties, etc.), and get a quantitative comparison of the performance of all methods in the database.

I realize that there are many difficulties in implementing this idea (chiefly: getting people to actually run all those problems, and measuring computational cost). But my question is: Does there exist such a database of results? (say in some particular subfield) or has an approach like this been implemented anywhere?

Both of the answers posted so far are about databases of problems. I'm asking about databases of results.

Answer 1

Of the criteria you've specified, I think the closest project that I know of would be the University of Florida sparse matrix collection. People routinely use this data set to compare sparse linear algebra solvers, and you can filter by application, number of nonzeros, dimensions of matrix, and so on with a really nice web interface, MATLAB interface, or Java GUI. I've seen tables of these problems listed in papers along with solver run time comparisons to 4 to 8 linear algebra solvers.

I agree that it would be useful to compile such databases, and furthermore, I think the UF sparse matrix collection approach for compiling the data is an excellent one, and would make a great start for anyone thinking about realizing this idea. Running all of the problems, in practice, doesn't seem like a major difficulty as long as you can get access to all of the solvers; if you have access to the solvers, and a reliable standard reference machine with all of the necessary software installed, then it should be a matter of running a script and collecting the data. The difficulty, in my mind, would be getting people to give you their software, if it isn't open source. If it's commercial, you could buy it, or even possibly get people to donate the software, which is something I know that happens in the optimization community when people are building portable software interfaces for commercial codes, such as in the COIN-OR project. But if it's research software that is neither commercial nor open source, then you need to convince people to buy into the endeavor, and they may not trust a third party to assess their software fairly.

I also know that in optimization, there are downloadable databases of problems (CUTEr comes to mind) and books of test problems for optimization. I have seen people (for instance, I'm specifically thinking of a talk by Ruth Misener at AIChE 2011) compare their optimization solver versus other solvers on databases of problems in presentations; I'm not sure what gets released publicly. I know that there's a tradition in optimization for comparison on a large scale (many solvers, many problems); I just don't think there's an online database available.

Another thing that I think is important is that we distinguish here between methods and software implementations. In scientific computing, we all talk about which methods are faster or slower based on things like computational complexity metrics, or our experiences with various problems. When it comes to measuring computational time quantitatively, however, unless one counts the number of FLOPs in a particular algorithm, one has to implement the algorithm in software and then measure the performance in some way (memory usage, wall clock time of execution, etc.). It makes sense to assess the performance of a method when looking at computational complexity or FLOP counts, because we don't need an implementation to measure such things, but the moment we're interested in actual wall clock run times, talking about methods is only useful as an abstract, colloquial device. (For instance, sparse direct solvers are fast on sufficiently small sparse problems with good fill-reducing heuristics, such as nested dissection.)

I bring up this distinction between methods and software because in such a database, I could also see the possibility of tracking the improvement in software over time. So, for instance, with something like, say, PETSc, or PyCLAW, or whatever software is being tested, it would be interesting to see what problems are affected positively (or negatively!) by upgrades in the software. This could be useful for researchers trying to decide if it's worth any potential costs in money and manpower to upgrade their codes. Another reason such a distinction is important is because a good method can be implemented badly; I think this possibility contributes to the reticence people sometimes have in sharing their research codes. A database of implementations would be able to distinguish between good and bad implementations of the same method (or types of methods).

I think whatever comes of this idea (and I hope something comes of it, and would be willing to contribute after my PhD), it's important to emphasize that distinction between software and methods, because if we're running test problems, we're going to be posting results for software.

Answer 2

There has been an attempt to create a list of "difficult" integrals to compute numerically (see papers by Walter Gautschi, Robert Piessens and others) but no formal database.

In the field of ODEs, there is the IVP test set, a test set for Initial Value Problems.

Answer 3

I'm not familiar with such a database, but I think it would be very difficult to define in a useful way. A better system, in my opinion, is to release the code in a runnable form so that the user can extend to test as comprehensively as they like. In terms of community repositories, the IVP test suite that GertVdE mentions also contains driver code to run several published ODE/DAE solvers on the test problems. In my publications, I try to release the code and all scripts used to create the analyze the output data and create the plots and tables appearing in the paper.

Answer 4

The Computational Aeroacoustics Workshop on Benchmark Problems has a similar goal in mind for aeroacoustic problems. They first present a set of benchmark problems ahead of time, and have people solve them using their own methods and compare the results. For example, here's the fourth meeting's proceedings: http://www.archive.org/details/nasa_techdoc_20040182258

Although, from what I can tell they do not compare how efficient their codes are against each other, they mainly look at accuracy.

Answer 5

Within earthquake modeling there has been several of these sorts of efforts. The ones that come to mind are the dynamic rupture simulation project and the source inversion project (http://eqsource.webfactional.com/wiki/). Both of these provides physical problems to solve and are agnostic (at some level) concerning the methods employed. Really these projects are about getting the domain specific modelers to compare with one another, not really a general set of test problems for every numerical pdes method. But you can compare with others results, and this has revealed a lot of problems with many codes.

One possibility to look at would be the madagascar project, which has been developed within the seismic imaging world but claims to be quite general purpose.