In my constrained problems (box constraints) I simply set my cost function to INFINITY (the c99 macro) if an inequality constraint is violated. This prevents the point being used, seems to work very nicely, and I don't need to come up with any arbitrary "magic numbers".

My code is used for gradient-free nonlinear optimization problems.

I get the impression that the penalty function approach must have been invented at a time when floating point mathematics would just "blow up", rather than handle INFINITY nicely.

Have I neglected something?

  • 3
    $\begingroup$ A closely related topic is regularization, which especially in statistics is often indispensable, And that for statistical, not numerical, reasons. $\endgroup$ Commented May 27 at 9:41
  • $\begingroup$ @StephanKolassa thanks for the link. Wow, looks like I have successfully sidestepped an "ocean of arbitrariness" by choosing my ultra-simple methods ;) $\endgroup$
    – m4r35n357
    Commented May 27 at 10:29

1 Answer 1


It is true that the penalty function method is quite old (Richard Courant, “Variational methods for the solution of problems of equilibrium and vibrations”, 1943), but your success with using INFINITY, or even a large constant magic number, has more to do with the particular non-gradient-based algorithm you are using. The penalty method is traditionally a way of converting a differentiable constrained optimization problem into a differentiable unconstrained problem having the same solution(s).

Using INFINITY (or a large constant) for all infeasible points requires, at the very least, an optimization algorithm that can tolerate discontinuities.

Also, it must be easy to find feasible points (points that satisfy the constraints), as is indeed the case when using box constraints. The large “flat” region where the objective function is constant or INFINITY gives no hint where a feasible point might lie.

Finally, your non-gradient-based algorithm might get into trouble if it uses a point at INFINITY in a calculation to decide where to search next. You are fortunate that the algorithm you are using does not do this, or is well-implemented so that it handles overflows and INFINITIES gracefully. An algorithm such as Nelder-Mead (for example) might fail, even if your problem is otherwise smooth.

  • 1
    $\begingroup$ Thanks for your answer. One of the methods I use is Nelder-Mead (with randomly placed large initial regular simplices). It is very tolerant of several vertices in the initial simplex being outside the feasible range. As long as there are not too many, the simplex just does the "right thing"! The other method I use is basically a guided "random" optimizer, so no gradients used, and the algorithm does not allow points to step outside the range in normal operation. If points are chosen outside the range they play no part until they are replaced on the next iteration! $\endgroup$
    – m4r35n357
    Commented May 27 at 8:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.