Is it pointless to use gradient based optimization algorithms if you can only provide a numerical gradient? If not, why provide a numerical gradient in the first place if it is trivial to perform finite differentiation for the optimization library itself?
Just to clarify, my question indeed is in a more general sense than a specific application. Although my field of application happens to be likelihood optimization under various statistical frameworks.
My issue with automatic differentiation is that there always seems to be a catch. Either the AD library can't propagate to external library calls (like BLAS) or you have to rework your workflow so drastically that it makes it a pain to deal with... especially if you're working with type sensitive languages. My gripes with AD are a separate issue altogether. But I want to believe!
I guess I need to better formulate my question but I'm doing a poor job of it. If have an option to either use a derivative-free optimization algorithm or a derivative based optimization algorithm with the caveat that I can only give it a numerical gradient, which one on average will be superior?