I implemented downhill simplex simulated annealing algorithm. Algorithm is very hard to tune, w.r.t. parameters including cooling schedule, starting temperature...
My first question is about convergence. In general, independently from choice of initial parameters, is it sure that algorithm converges to global optimal solution?
More precisely: I use SA with downhill simplex based on Num.recipes, however, I am adding some specific actions for observing constraints. Do you think my additional conditions for constraints are cause for bad convergence?
Then, I would like your advice on how to go on with solver conception: for instance, my algo is tested on data in 10-D space (10 parameters). It goes as follows: a preprocessing (monte carlo) picks up a good starting point. Then I run simplex downhill --> does not converge. I run simulated annealing (with simplex downhill to go to next step), and it remains far to solution (RMSE == 0.001). Excel solver on these data does the job very good. Excel method is generalized reduced gradient. Should i go on with fine tuning for SA, or go to non-derivative-free methods s.a. GRG ?
Finally, to speak the same language: with my example, convergence is not reach AT ALL. i don't have time to reach a local minimum either, since I have put some exit conditions when the simplex gets stuck. what does one mean in general with 'solver does not converge'? does it mean that termination condition (s.a. go below a tolerance value) is met and solution is sub optimal, or can it mean that termination condition is never met?
Thanks & regards.
EDIT
My objective function is given explicitly. It can, for instance, look like
$F(p) = \sum_{j=1}^{n}\Bigg(y_{j} - (p_{1} + p_{2}\exp{\Big(-\frac{(x_{i}-p_{3})^{2}}{p_{4}}\Big)} + p_{5}\exp{\Big(-\frac{(x_{j}-p_{6})^{2}}{p_{7}}\Big)} + p_{8}\exp{\Big(-\frac{(x_{j}-p_{9})^{2}}{p_{10}}\Big))}\Bigg)^{2}$
so, the squared residuals with a 'three Gaussians + step' model.
This is an example, solver should solve for any combination of functions among gaussians, exponential, power, polynomial, linear, rational with 'reasonable' number of parameters.
Box constraints on parameters are specified, with form $l_{i} \leq p_{i} \leq u_{i}$.