I asked this question earlier on StackOverflow, but it's obviously better suited for SciComp:
While there seem to be lots of references online which compare automatic differentiation methods and frameworks against each other, I can't seem to find anything on how I should expect automatic differentiation to compare to hand-written derivative evaluation.
My particular situation: I'm doing a gradient-descent of a scalar cost function, with a few thousand variables and significant (>99%) sparsity. I have a C++ function to compute this cost. I've also manually derived and implemented functions to compute the gradient and Hessian. (The cost function and its derivatives are not tuned in any way, they're just a straightforward implementation.) A typical descent to a minimum takes about ten seconds.
I want to more easily use different cost functions, and using AD seems like a good way to proceed. I used CppAD to automatically evaluate the gradient and Hessian; these values are exactly the same as those from my manual derivative functions. But using AD, a typical gradient descent takes about five minutes.
I was surprised at how much slower AD was, but then realized that I don't really know how fast it should be. I'm not looking so much for "How do I speed up my computations?", as "What should I expect?"
How much slower should automatic differentiation be, compared to hand-written derivatives? How about for first- and second-order derivatives?