If we optimize some parameter using 4 optimization algorithms, 2 of which are population based (say A and B) and 2 trajectory methods (single point search)(say C and D); what statistical test can be used for comparing 4 of these. The initial population of A and B is the same, so is for C and D. However, population size of A and B is $n$ and the population size of C and D is 1. So, I think, we should use paired t-test for comparing (A and B) and (C and D). And should use unpaired t-test for all other combinations with Bonferroni correction. But what should actually be done?
$\begingroup$
$\endgroup$
5
-
3$\begingroup$ Your question is not clear enough to provide much in terms of answers. For example, what do you want to compare? The quality of the answers each of the four algorithms provides? The run time? The numerical cost? $\endgroup$– Wolfgang BangerthOct 29, 2014 at 13:35
-
1$\begingroup$ the numerical cost. The algorithms have been run 20 times for 10 functions. I want to compare on the basis of objective function value, i.e. cost $\endgroup$– megamindOct 29, 2014 at 14:22
-
$\begingroup$ If you are asking a statistics question that would have the same answer if "objective function values produced by these optimization algorithms" is replaced by "qualities of widgets produced by these factories" then you are asking on the wrong forum. It would be a question about statistical hypothesis testing not about optimization algorithms or about widget production. $\endgroup$– k20Oct 29, 2014 at 21:00
-
$\begingroup$ I don't understand your answer to my question. You say "numerical cost" and "objective function value", but these are different things. Numerical cost is the number of function evaluations (or the number of CPU instructions) before the algorithm terminates, whereas the objective function value is the value the program returns. These are different things. $\endgroup$– Wolfgang BangerthOct 30, 2014 at 2:45
-
$\begingroup$ the number of function evaluations are same for all algos. I want to compare on the basis of minimum objective function value obtained for each algo. $\endgroup$– megamindNov 2, 2014 at 7:04
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Established methodologies for benchmarking optimization software can be found in publications such as Benchmarking Optimization Software with Performance Profiles, Benchmarking Derivative-Free Optimization Algorithms, and Derivative-free optimization: a review of algorithms and comparison of software implementations.
Generally speaking, algorithms are benchmarked against a suite of problems, and then performance profiles are constructed based on whether or not the algorithm successfully solved each problem to within a given tolerance and time limit. $t$-tests are not typically used.
answered Oct 31, 2014 at 18:34
$\begingroup$
$\endgroup$
An interesting method to find "best optimization algorithms" for a class of test instances is the iRace (resp. F-Race) approach, for which an R-package exists.
