If we optimize some parameter using 4 optimization algorithms, 2 of which are population based (say A and B) and 2 trajectory methods (single point search)(say C and D); what statistical test can be used for comparing 4 of these. The initial population of A and B is the same, so is for C and D. However, population size of A and B is $n$ and the population size of C and D is 1. So, I think, we should use paired t-test for comparing (A and B) and (C and D). And should use unpaired t-test for all other combinations with Bonferroni correction. But what should actually be done?
Established methodologies for benchmarking optimization software can be found in publications such as Benchmarking Optimization Software with Performance Profiles, Benchmarking Derivative-Free Optimization Algorithms, and Derivative-free optimization: a review of algorithms and comparison of software implementations.
Generally speaking, algorithms are benchmarked against a suite of problems, and then performance profiles are constructed based on whether or not the algorithm successfully solved each problem to within a given tolerance and time limit. $t$-tests are not typically used.
An interesting method to find "best optimization algorithms" for a class of test instances is the iRace (resp. F-Race) approach, for which an R-package exists.