There seem to be two main kinds of test function for no-derivative optimizers:
- one-liners like the Rosenbrock function ff., with start points
- sets of real data points, with an interpolator
Is it possible to compare say 10d Rosenbrock
with any real 10d problems ?
One could compare in various ways: describe the structure of local minima,
or run optimizers A B C on Rosenbrock and on some real problems;
but both of these seem difficult.
(Maybe theorists and experimenters are just two quite different cultures, so I'm asking for a chimera ?)
- scicomp.SE question: Where can one obtain good data sets/test problems for testing algorithms/routines?
- Hooker, "Testing Heuristics: We Have It All Wrong" is scathing: "the emphasis on competition ... tells us which algorithms are better but not why."
(Added in September 2014):
The plot below compares 3 DFO algorithms on 14 test functions in 8d from 10 random start points: BOBYQA PRAXIS SBPLX from NLOpt
$\times$ 14 N-dimensional test functions, Python under gist.github from this Matlab by A. Hedar
$\times$ 10 uniform-random startpoints in each function's bounding box.
On Ackley, for example, the top row shows that SBPLX is best and PRAXIS terrible; on Schwefel, the bottom right panel shows SBPLX finding a minimum on the 5 th random start point.
Overall, BOBYQA is best on 1, PRAXIS on 5, and SBPLX (~ Nelder-Mead with restarts) on 7 of 13 test functions, with Powersum a tossup. YMMV ! In particular, Johnson says, "I would advise you not to use function-value (ftol) or parameter tolerances (xtol) in global optimization."
Conclusion: don't put all your money on one horse, or on one test function.