Techniques to remove a function from Levenberg-Marquardt when it is against box constraints

Question

I have a somewhat large (20+ dimensional) root finding problem that I'm solving with Levenberg-Marquardt. One of the functions has box constraints on [0, 2]. When it is against those bounds it will not be able to find a zero of the function. Leaving that in the solution and letting the optimizer try to find a minimum does not find a good solution for the rest of the problem.

I've tried a few different techniques to insert zeros into that function, the most naive being:

if ( y0[m] <= 0 || y0[m] >= 2 )
  z[n] = 0
else
  z[n] = H0tf
end

(where y0[m] is the independent variable that is bounded, z[n] is the zero function being produce, and H0tf is a somewhat complicated function).

I've also tried less naive ways of introducing those zeros, with various potentials introduced near 0 and 2 so that the function becomes smooth and not discontinuous.

The overall difficulty of all of those approaches is that now in the non-bounded case the optimizer needs to solve a global optimization problem where there may be a zero at 0 and 2 and also the correct 0 in H0tf somewhere in the [0, 2] range, which gets very 'hacky' trying to solve that with a local optimizer.

Is there a better way to handle somehow deleting this function from consideration of the optimizer when it is at its bounds? And what do you call such a technique?

Answer 1

Did you try adding penalty regularisation scheme to your least-squares objective function?

The Levenburg-Marquardt algorithm with attempt to minimise the residual, $r(x_i)$ where $x_i$ is your parameter vector. You could try adding a penalty term which penalizes the solver for suggesting to place the troublesome parameter near to a boundary.

$r^\prime(x_i) = r(x_i) + p(x_i)$

Here $p(x_i)$ is the penalty term, which will be non-zero when one of your parameters gets stuck or approaches the boundary. Obviously, you should only accept solutions where $p(x_i)=0$.

As an aside, have you tried Trust Region Reflection algorithm. These things can be problem specific, but I have been very impressed with how efficient and robust this can be. It is the default option in scipy.