Are penalty functions still "necessary"?

Question

In my constrained problems (box constraints) I simply set my cost function to INFINITY (the c99 macro) if an inequality constraint is violated. This prevents the point being used, seems to work very nicely, and I don't need to come up with any arbitrary "magic numbers".

My code is used for gradient-free nonlinear optimization problems.

I get the impression that the penalty function approach must have been invented at a time when floating point mathematics would just "blow up", rather than handle INFINITY nicely.

Have I neglected something?

Answer 1

It is true that the penalty function method is quite old (Richard Courant, “Variational methods for the solution of problems of equilibrium and vibrations”, 1943), but your success with using INFINITY, or even a large constant magic number, has more to do with the particular non-gradient-based algorithm you are using. The penalty method is traditionally a way of converting a differentiable constrained optimization problem into a differentiable unconstrained problem having the same solution(s).

Using INFINITY (or a large constant) for all infeasible points requires, at the very least, an optimization algorithm that can tolerate discontinuities.

Also, it must be easy to find feasible points (points that satisfy the constraints), as is indeed the case when using box constraints. The large “flat” region where the objective function is constant or INFINITY gives no hint where a feasible point might lie.

Finally, your non-gradient-based algorithm might get into trouble if it uses a point at INFINITY in a calculation to decide where to search next. You are fortunate that the algorithm you are using does not do this, or is well-implemented so that it handles overflows and INFINITIES gracefully. An algorithm such as Nelder-Mead (for example) might fail, even if your problem is otherwise smooth.