Optimization with order constraints on parameters

Question

Are there optimizers where it is possible to specify ordinal ranking of parameters?

Assume that I have a function of three parameters $f(\theta_1, \theta_2, \theta_3)$. Are there optimizers such that I can specify $$ {\arg\min}_{\theta_1 > \theta_2 >\theta_3}f(\theta_1, \theta_2, \theta_3) $$

Assume that $f$ is smooth ($n$-th order differentiable in each of the parameters).

Answer 1

An approach I often use when applying unconstrained optimisation algorithms to constrained problems is to transform the parameter space such that the constraints cannot be violated.

For your problem I would define

$$ g(x_1, x_2, x_3) = f(x_1, x_1-x_2^2, x_1-x_2^2-x_3^2) $$

then solve

$$ \arg \min g(x_1, x_2, x_3) $$

and finally recover the solution to the original problem with

$$ \begin{align*} \theta^*_1 &= x^*_1\\ \theta^*_2 &= x^*_1 - x^{*2}_2\\ \theta^*_3 &= x^*_1 - x^{*2}_2 - x^{*2}_3 \end{align*} $$

Of course this results in $\theta^*_1\geq\theta^*_2\geq\theta^*_3$ which isn't quite what you asked for. To get a strict ranking you'll need to bump $x_1-x^2_2$ and $x_1-x^2_2-x^2_3$ down at the last digit of precision.

Answer 2

If you relax the strictness criterion, then yes, you could pose your question as a constrained optimization problem:

\begin{align} & \min_{\theta_{1}, \theta_{2}, \theta_{3}} f(\theta_{1}, \theta_{2}, \theta_{3}) \\ \textrm{s.t.} & \theta_{1} \geq \theta_{2} \geq \theta_{3}. \end{align}

To approximate $\theta_{1} > \theta_{2} > \theta_{3}$, you could add a parameter $\varepsilon > 0$ such that:

\begin{align} & \min_{\theta_{1}, \theta_{2}, \theta_{3}} f(\theta_{1}, \theta_{2}, \theta_{3}) \\ \textrm{s.t.} & \theta_{1} \geq \theta_{2} + \varepsilon \\ & \theta_{2} \geq \theta_{3} + \varepsilon \end{align}

These variants of your constraints are linear, so provided that your function $f$ is well-behaved (smooth, easy to calculate, easy to compute derivatives, derivatives are well-conditioned, etc.), any constrained optimization solver should be able to solve your problem without issue.