10
$\begingroup$

I am just learning (more) about automatic differentiation (AD) and at this stage it kind of seems like black magic to me. The second paragraph of its Wikipedia article makes it sound too good to be true: it is extremely fast and is exact (no round off, no discretisation). I am left wondering why finite difference (FD) is so ubiquitous in scientific computing. Looking this up, I seem to only find tutorials on how to implement AD, the advantages of AD, and its applications in gradient-based optimisers. But what is an example of when not to use AD, and instead use FD? Surely there must many.

As just one example, in computational electromagnetics a FD approach is very standard; why can we not propagate Maxwell's equations with AD (FDTD: why not ADTD?)? It is clearly not because the developers aren't aware of it because the same people implement AD for inverse design purposes (why AD instead of FD for inverse design?). Naively, to me it seems like having an exact derivative should be more important when propagating Maxwell's equations than when taking the derivative of an objective function.

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ All floating point computations involve round off and AD doesn’t magically get around this. What the Wikipedia article is getting at is that FD approximations can lead to very large round off errors due to so-called catastrophic cancellation where significant accuracy is lost by subtracting two nearly equal numbers. $\endgroup$
    – eepperly16
    Commented Jul 27, 2020 at 15:49

1 Answer 1

Reset to default
14
$\begingroup$

Given code that computes a function $f(x)$, automatic differentiation tools produce a code that can compute $f(x)$ and its derivatives at the same time. Solving a differential equation is an entirely different problem and AD doesn't solve differential equations (although AD tools are sometimes useful in connection with PDE constrained optimization.)

AD tools are quite good at computing derivatives and should be used more often. However, there are circumstances where they simply can't be used. The most common reason that AD can't be used is that the code you have is a "black-box" for which source code is not available. Some other codes are so large and complicated that AD tools simply fail to handle them.

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ Ah, in hindsight that is actually pretty obvious, thanks. So with the exception of monstrously large functions, AD is basically always preferred to FD if we have the function f(x)? $\endgroup$
    – tmph
    Commented Jul 25, 2020 at 23:25
  • 2
    $\begingroup$ If you have code that implements $f(x)$, and that code is not too ugly, than AD tools are a good choice. However, if the source code isn't available or if the code is too complicated for your AD tools, then finite difference derivatives are generally the way to go. $\endgroup$
    – Brian Borchers
    Commented Jul 26, 2020 at 13:58
  • 1
    $\begingroup$ @tmph I agree with this but want to expand on two things: 1. Yes AD does not fundamentally help with solving differential equations; DE's are literally the derivatives of $f$ given in closed form, and the question is to determine $f$ given (a relation on) its derivatives. I want to add that AD is a good way to get other derivatives in various numerical methods meant to solve DE's, e.g. shooting and collocation methods which are often formulated as optimization problems and need gradients of a suitable loss function. This was hinted at in mentioning "PDE constrained optimization." $\endgroup$
    – jnez71
    Commented Jul 27, 2020 at 17:19
  • 2
    $\begingroup$ 2. AD cannot leverage sparsity (neither can FD). This is where symbolic differentiation (which can be automated!) really shines. I have worked on problems that required the linearization of very high dimensional functions with huge but sparse Jacobians, and used symbolic tools to be sure I only compute the nonzero entries. AD (and in some sense FD) will not be able to "see ahead of runtime" the cancellation in e.g. $\frac{\partial f_{36}}{\partial x_{72}} = \cos(x_{20})^2 + \sin(x_{20})^2 - 1 =0$. Hopefully that added insight! $\endgroup$
    – jnez71
    Commented Jul 27, 2020 at 17:56
  • 1
    $\begingroup$ So yeah, even though AD is "exact" like SD rather than approximate like FD, there are often reductions SD can make (if done carefully) that aren't deduced from the AD graph. On the other hand, given $f$ source code, SD can be a lot of work while AD is super convenient. In any case, I think it's important to distinguish between FD to approximate derivatives and FD to discretize a DE. Though mathematically identical, the purpose of FD in discretization is better seen from the variational perspective: FDM just proposes a piecewise-linear $f$ with knots at every point on a rectangular grid. $\endgroup$
    – jnez71
    Commented Jul 27, 2020 at 17:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.