I've developed a derivative-free optimization method, looking for comments

Question

Here is the URL: https://github.com/avaneev/biteopt

I've tested it on numerous global optimization benchmarking functions (included), and on real-world hyperparameter optimization problems I have. Seems to be working quite well except in comparison to deterministic methods it's necessary to make several attempts at different random seeds, so the iteration budget may be high. But stochasticity of this method gives a chance to solve a problem which can't be sufficiently solved by deterministic methods. Anyway, most benchmarking functions solve in 1 attempt given enough iteration budget.

Works best for non-convex problems, and can also solve convex problems, but of course slower than deterministic methods. Can also solve non-linear constrained problems, but such constraints increase convergence time considerably (though, this application was not tested thoroughly).

Still working on the method.

I would like to hear comments from users that have some practical models (e.g. black-box hyperparameter optimization) which are still needed to be solved acceptably - whether this method works or not for their models, possibly with the description of the model.

Here is the description of the method. The algorithm consists of the following elements:

1. A cost-ordered population of previous solutions is maintained. A solution is an independent parameter vector which is evolved towards a better solution. On every iteration, the best solution is evolved. $$x_\text{new}=x_\text{best}$$ Below, i is either equal to rand(1, N) or in the range [1; N], depending on the AllpProb probability. Probabilities are defined in the range [0; 1] and in many instances in the code were replaced with simple resetting counters for more efficiency. Parameter values are internally normalized to [0; 1] range and, to stay in this range, are wrapped in a special manner before each function evaluation. Algorithm's hyper-parameters (probabilities) were pre-selected and should not be changed.

2. Depending on the RandProb probability, a single (or all) parameter value randomization is performed using "bitmask inversion" operation. $$mask= 2^{1+\lfloor(0.999999997-rand(0\ldots1)^4 )\cdot MantSize\rfloor}-1$$ $$MantMult=2^{MantSize}$$ $$x_\text{new}[i] = \frac{\lfloor x_\text{new}[i]\cdot MantMult \rfloor \bigotimes mask }{MantMult}$$ Plus, with CentProb probability the random "step in the right direction" operation is performed using the centroid vector, twice. $$m_1=\text{rand}(0\ldots1)\cdot CentSpan$$ $$x_\text{new}[i]=x_\text{new}[i]-m_1(x_\text{new}[i]-x_\text{cent}[i])$$ $$m_2=\text{rand}(0\ldots1)\cdot CentSpan$$ $$x_\text{new}[i]=x_\text{new}[i]-m_2(x_\text{new}[i]-x_\text{cent}[i])$$ With RandProb2 probability an alternative randomization method is used. $$x_\text{new}[i]=x_\text{new}[i]+(-1)^{s}(x_\text{cent}[i]-x_\text{new}[i]), \quad i=1,\ldots,N,\\ \quad s\in\{1,2\}=(\text{rand}(0\ldots1)<0.5 ? 1:2)$$

3. (Not together with N.2) the "step in the right direction" operation is performed using the random previous solution, current best and worst solutions. This is conceptually similar to Differential Evolution's "mutation" operation. $$x_\text{new}=x_\text{best}-\frac{(x_\text{worst}-x_\text{rand})}{2}$$

4. With ScutProb probability a "short-cut" parameter vector change operation is performed. $$z=x_\text{new}[\text{rand}(1,N)]$$ $$x_\text{new}[i]=z, \quad i=1,\ldots,N$$

5. After each objective function evaluation, the highest-cost previous solution is replaced using the cost constraint.

You can find this algorithm implemented in the optimize() function in biteopt.h on lines 284-395, it does not involve any higher-order math.

Answer 1

I would like to hear comments from users that have some practical models (e.g. black-box hyperparameter optimization) which are still needed to be solved acceptably - whether this method works or not for their models, possibly with the description of the model.

Looks like you want somebody to invest what may be considerable time and energy in trying out your new method on their models, and report on the results.

As a modeler, I have to ask myself the question: why would I do that? It'd take me some time to change my code and link in your method. What does your optimization method have to offer that may be better than my current method of choice? Now, we can't possibly answer this question because (a) you don't know what models I'm working with, and (b) I don't know what kind of problems your method is good for - beyond some "real-life optimization problems" where it "seems to be working quite well".

Bottom line is you have to do a heck of a better job "selling" your method to the community at large. You could start by:

• testing the method on a representative sample of problems where you expect it to do well, and report on the results;
• compare the results with those achieved by similar, known methods (to show us that you're at least competitive with the state of the art);
• show that your method converges, and under which conditions - if you can't come up with a formal proof, then at least give some heuristic arguments (backed up by some solid experimental data);
• advise on the choice of hyper-parameters, like RandProb, CentProb, etc.
• tell us what problems we should not be using your method on, and why.