I have a function of many variables (~200-2000) which I am optimizing with some success using L-BFGS. While the function is expensive to evaluate, the gradient can be computed with not much additional effort.
I would like to know if there are any techniques which could allow this optimization to be sped up by parallel evaluation of the function. On this problem, L-BFGS averages ~1.1-1.3 evaluations per iteration, so I think to a good approximation, I can ignore the line search component of the algorithm. From this perspective then, on every iteration, L-BFGS chooses a tentative input evaluates the function and its gradient, updates its estimation of the inverse Hessian and repeats.
I imagine it is possible to enhance this approach by choosing multiple tentative inputs and using the multiple resulting gradients to form a better estimation of the Hessian. I imagine there is diminishing returns of some sort, the point selected by L-BFGS probably has the highest expected information, but surely there is some additional information about the Hessian that one can infer by using more function evaluations per iteration.
Has this type of approach been investigated? I know there are many resources for parallel gradient-free optimization, as well as parallel gradient estimation, but I have not come across anything like this yet.