# Testing numerical optimization methods: Rosenbrock vs. real test functions

There seem to be two main kinds of test function for no-derivative optimizers:

• one-liners like the Rosenbrock function ff., with start points
• sets of real data points, with an interpolator

Is it possible to compare say 10d Rosenbrock with any real 10d problems ?
One could compare in various ways: describe the structure of local minima,
or run optimizers A B C on Rosenbrock and on some real problems;
but both of these seem difficult.

(Maybe theorists and experimenters are just two quite different cultures, so I'm asking for a chimera ?)

The plot below compares 3 DFO algorithms on 14 test functions in 8d from 10 random start points: BOBYQA PRAXIS SBPLX from NLOpt
$\times$ 14 N-dimensional test functions, Python under gist.github from this Matlab by A. Hedar
$\times$ 10 uniform-random startpoints in each function's bounding box.

On Ackley, for example, the top row shows that SBPLX is best and PRAXIS terrible; on Schwefel, the bottom right panel shows SBPLX finding a minimum on the 5 th random start point.

Overall, BOBYQA is best on 1, PRAXIS on 5, and SBPLX (~ Nelder-Mead with restarts) on 7 of 13 test functions, with Powersum a tossup. YMMV ! In particular, Johnson says, "I would advise you not to use function-value (ftol) or parameter tolerances (xtol) in global optimization."

Conclusion: don't put all your money on one horse, or on one test function.

-

Simple functions like Rosenbrock's are used to debug and pre-test newly written algorithms: They are fast to implement and to execute, and a method that cannot solve the standard problems well is unlikely to work well on real life problems.

For a recent thorough comparison of derivative-free methods for expensive functions, see Derivative-free optimization: A review of algorithms and comparison of software implementations. L.M. Rios, N.V. Sahinidis - doi 10.1007/s10898-012-9951-y Journal of Global Optimization, 2012. (See also the accompanying webpage: http://archimedes.cheme.cmu.edu/?q=dfocomp)

-
Prof. Neumaier, could you point to some real problems, evidence, for "a method that cannot solve the standard problems well is unlikely to work well on real life problems" ? I realize that that's not easy. (I'd be interested in your comments on Hooker.) Also, a quick look at c models from your link shows princetonlibgloballib requires AMPL, and source_convexmodels *.c all have a missing ";" after fscanf() -- trivial but –  denis Aug 11 '12 at 11:07
@Denis: Problems like Rosenbrock stem from the early days of automated optimization, where people isolated the typical difficulties in simple representative examples that can be studied without the numerical complexities of real-life problems. Thus they are not really artificial, but simplified models of real difficulties. For example, Rosenbrock illustrates the combined effect of strong nonlinearity and mild ill-condition. –  Arnold Neumaier Aug 11 '12 at 11:14
The AMPL site ampl.com offers a free student version for AMPL. –  Arnold Neumaier Aug 11 '12 at 11:15

The advantage of synthetic testcases like the Rosenbrock function is that there is existing literature to compare with, and there is a sense in the community how good methods behave on such testcases. If everyone used their own testcase it would be much harder to come to a consensus which methods work and which don't.

-

You can have the best of both worlds. NIST has a set of problems for minimizers, like fitting this 10th degree polinomial, with expected results and uncertainties. Of course, proving that these values are the actual best solution, or the existence and properties of other local minima is more difficult than on a controlled mathematical expression.

-
Well, the NIST problems are small (2 3 1 1 11 7 6 6 6 6 6 params). Are there test sets that are both "real" and reproducible, for any corner of "real" ? Cf. a Request for Simulation-Based Optimization Problems –  denis May 13 at 13:12