# Testing numerical optimization methods: Rosenbrock vs. real test functions

There seem to be two main kinds of test function for no-derivative optimizers:

• one-liners like the Rosenbrock function ff., with start points
• sets of real data points, with an interpolator

Is it possible to compare say 10d Rosenbrock with any real 10d problems ?
One could compare in various ways: describe the structure of local minima,
or run optimizers A B C on Rosenbrock and on some real problems;
but both of these seem difficult.

(Maybe theorists and experimenters are just two quite different cultures, so I'm asking for a chimera ?)

The plot below compares 3 DFO algorithms on 14 test functions in 8d from 10 random start points: BOBYQA PRAXIS SBPLX from NLOpt
$\times$ 14 N-dimensional test functions, Python under gist.github from this Matlab by A. Hedar
$\times$ 10 uniform-random startpoints in each function's bounding box.

On Ackley, for example, the top row shows that SBPLX is best and PRAXIS terrible; on Schwefel, the bottom right panel shows SBPLX finding a minimum on the 5 th random start point.

Overall, BOBYQA is best on 1, PRAXIS on 5, and SBPLX (~ Nelder-Mead with restarts) on 7 of 13 test functions, with Powersum a tossup. YMMV ! In particular, Johnson says, "I would advise you not to use function-value (ftol) or parameter tolerances (xtol) in global optimization."

Conclusion: don't put all your money on one horse, or on one test function.

-

Simple functions like Rosenbrock's are used to debug and pre-test newly written algorithms: They are fast to implement and to execute, and a method that cannot solve the standard problems well is unlikely to work well on real life problems.

For a recent thorough comparison of derivative-free methods for expensive functions, see Derivative-free optimization: A review of algorithms and comparison of software implementations. L.M. Rios, N.V. Sahinidis - doi 10.1007/s10898-012-9951-y Journal of Global Optimization, 2012. (See also the accompanying webpage: http://archimedes.cheme.cmu.edu/?q=dfocomp)

-
Prof. Neumaier, could you point to some real problems, evidence, for "a method that cannot solve the standard problems well is unlikely to work well on real life problems" ? I realize that that's not easy. (I'd be interested in your comments on Hooker.) Also, a quick look at c models from your link shows princetonlibgloballib requires AMPL, and source_convexmodels *.c all have a missing ";" after fscanf() -- trivial but – denis Aug 11 '12 at 11:07
@Denis: Problems like Rosenbrock stem from the early days of automated optimization, where people isolated the typical difficulties in simple representative examples that can be studied without the numerical complexities of real-life problems. Thus they are not really artificial, but simplified models of real difficulties. For example, Rosenbrock illustrates the combined effect of strong nonlinearity and mild ill-condition. – Arnold Neumaier Aug 11 '12 at 11:14
The AMPL site ampl.com offers a free student version for AMPL. – Arnold Neumaier Aug 11 '12 at 11:15

The advantage of synthetic testcases like the Rosenbrock function is that there is existing literature to compare with, and there is a sense in the community how good methods behave on such testcases. If everyone used their own testcase it would be much harder to come to a consensus which methods work and which don't.

-

(I hope there is no objection to my tacking onto the end of this discussion. I'm new here, so please let me know if I have transgressed!)

Test functions for evolutionary algorithms are now much more complicated than they were even 2 or 3 years ago, as can be seen by the suites used in competitions at conferences like the (very recent) 2015 Congress on Evolutionary Computation. See:

http://www.cec2015.org/

These test suites now include functions with several non-linear interactions between variables. The number of variables can be as large as 1000, and I would guess that might increase in the near future.

Another very recent innovation is a "Black Box Optimization Competition". See: http://bbcomp.ini.rub.de/

An algorithm can query the value f(x) for a point x, but it does not obtain gradient information, and in particular it cannot make any assumptions on the analytic form of the objective function.

In a sense, this might be closer to what you referred to as a "real problem" but in an organised, objective setting.

-
1) "no objection": on the contrary, your good links are welcome ! 2) any good plots there ? Methods and problems are both fractalizing, so it's getting harder and harder for anyone to find a problem like theirs. In particular, would you know of methods for time series forecasting ? – denis Jun 10 '15 at 9:31
The objective functions for the CEC 2015 Competition on Dynamic Multi-Objective Optimization can be seen at: sites.google.com/site/cec2015dmoocomp/competition-process/… For other competitions, go to cec2015.org and click on competitions, then click on accepted competitions. Each one has its own functions. Papers on some of them have lovely plots (for the 2D cases). GECCO conference competitions can be found at: sigevo.org/gecco-2015/competitions.html#bbc Results will be available after July 15. – Lysistrata Jun 10 '15 at 12:34

You can have the best of both worlds. NIST has a set of problems for minimizers, like fitting this 10th degree polinomial, with expected results and uncertainties. Of course, proving that these values are the actual best solution, or the existence and properties of other local minima is more difficult than on a controlled mathematical expression.

-
Well, the NIST problems are small (2 3 1 1 11 7 6 6 6 6 6 params). Are there test sets that are both "real" and reproducible, for any corner of "real" ? Cf. a Request for Simulation-Based Optimization Problems – denis May 13 '14 at 13:12