I am using nonlinear optimizers such as BOBYQA to train a model with 10-20 parameters. It so happens that some of the parameters have high correlation. Roughly speaking, imagine that you are fitting parameters $a,b$ in a function like this:


The response to the change in parameters $a,b$ is much larger than the response in to the change of $a-b$. Would performance of Powell's nonlinear optimization algorithms such as BOBYQA, NEWUOA, etc., improve if I would "rotate parameters" to isolate gross responses from fine ones?

That is, would nonlinear optimizers perform better if instead of optimizing $a,b$ in the above function I would optimize for parameters $\alpha,\beta$ a function like this:


The real model is more complicated, of course, and I'd like to find out whether changing parameters to minimize correlations has a chance to improve anything.


If possible, you want to choose a well-scaled objective for your problem, which would mean de-correlating your parameters as much as possible.

For a quadratic approximation to an objective function, correlated parameters correspond to objective function isosurfaces that are ellipsoidal in shape; the ideal shape for isosurfaces would be roughly spherical. In algebraic terms, ellipsoidal isosurfaces correspond to ill-conditioned Hessian matrices (the relationship being that the magnitude of the eigenvalues of the Hessian corresponds to the lengths of the semiaxes of the ellipsoids), and for second-order methods, an ill-conditioned Hessian definitely degrades convergence. Empirically, this behavior would be seen by taking small "zig-zag" steps as one approaches a local optimum; from a numerical analysis standpoint, it follows by noting that second-order methods take a form that looks roughly like

\begin{align} H(x^{n})(x^{n+1} - x^{n}) = -\nabla{f}(x^{n}), \end{align}

where $H$ is the Hessian, $f$ is the objective function, and $\nabla{f}$ is the gradient of the objective function, yielding an ill-conditioned linear system that will be solved inaccurately with numerical methods. (Here, I've assumed a Newton method, but quasi-Newton methods operate similarly.)

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.