Timeline for When and when not to use automatic differentiation
Current License: CC BY-SA 4.0
8 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jul 27, 2020 at 17:56 | comment | added | jnez71 | So yeah, even though AD is "exact" like SD rather than approximate like FD, there are often reductions SD can make (if done carefully) that aren't deduced from the AD graph. On the other hand, given $f$ source code, SD can be a lot of work while AD is super convenient. In any case, I think it's important to distinguish between FD to approximate derivatives and FD to discretize a DE. Though mathematically identical, the purpose of FD in discretization is better seen from the variational perspective: FDM just proposes a piecewise-linear $f$ with knots at every point on a rectangular grid. | |
| Jul 27, 2020 at 17:56 | comment | added | jnez71 | 2. AD cannot leverage sparsity (neither can FD). This is where symbolic differentiation (which can be automated!) really shines. I have worked on problems that required the linearization of very high dimensional functions with huge but sparse Jacobians, and used symbolic tools to be sure I only compute the nonzero entries. AD (and in some sense FD) will not be able to "see ahead of runtime" the cancellation in e.g. $\frac{\partial f_{36}}{\partial x_{72}} = \cos(x_{20})^2 + \sin(x_{20})^2 - 1 =0$. Hopefully that added insight! | |
| Jul 27, 2020 at 17:19 | comment | added | jnez71 | @tmph I agree with this but want to expand on two things: 1. Yes AD does not fundamentally help with solving differential equations; DE's are literally the derivatives of $f$ given in closed form, and the question is to determine $f$ given (a relation on) its derivatives. I want to add that AD is a good way to get other derivatives in various numerical methods meant to solve DE's, e.g. shooting and collocation methods which are often formulated as optimization problems and need gradients of a suitable loss function. This was hinted at in mentioning "PDE constrained optimization." | |
| Jul 26, 2020 at 17:50 | vote | accept | tmph | ||
| Jul 26, 2020 at 13:58 | history | edited | Brian Borchers | CC BY-SA 4.0 |
added 3 characters in body
|
| Jul 26, 2020 at 13:58 | comment | added | Brian Borchers | If you have code that implements $f(x)$, and that code is not too ugly, than AD tools are a good choice. However, if the source code isn't available or if the code is too complicated for your AD tools, then finite difference derivatives are generally the way to go. | |
| Jul 25, 2020 at 23:25 | comment | added | tmph | Ah, in hindsight that is actually pretty obvious, thanks. So with the exception of monstrously large functions, AD is basically always preferred to FD if we have the function f(x)? | |
| Jul 25, 2020 at 22:39 | history | answered | Brian Borchers | CC BY-SA 4.0 |