I have been looking at automatic differentiation for solving differential equations lately. I understand the basic ideas of using Dual numbers and such for finding derivatives, etc. However, I feel like I am not understanding the geometric interpretation of automatic differentiation and dual numbers.

Let's consider a simple differential question or system of differential equations.

$$ \frac{dy}{dt} = f(y, t) $$

So let me set a little context first for how I understand the application of automatic differentiation to numerically solving differential equations. If I make any mistakes, please correct me. So if you have a relatively low dimensional system with reasonable time-scale separation, then you could get away with some second order explicit or RK type solvers. That should provide good accuracy and fast compute time. In such cases, you don't need use automatic differentiation, because you can compute the derivative f(y,t) directly--if the solver is explicit.

It seems that AD become more relevant for larger systems of equations where there is less time-scale separate and more stiffness. In those cases, implicit solvers are the best approach to take. But when using an implicit solver you have to solve a root finding problem using something like Newton's method or Anderson Acceleration, etc.

So in Newton's method we have to compute the jacobian of the original differential equation $f(y,t)$. AD become relevant or useful for computing the jacobian of the original RHS of the differential equation when finding the roots for the implicit equation.

It seems like the older approach was to use numerical differentiation or symbolic methods to represent the jacobian in the root finding implementation. And in the case of numerical differentiation, this is susceptible to floating point errors and is generally ill conditioned. And in the symbolic case, this is generally slow to solve.

So that is my understanding of why AD is used.

BUT, in many of the explanation of AD, the authors will explain the reason for using AD, but then will dive deep into algebraic constructions of Dual numbers and Dual number arithmetic. So I don't have a sense of the geometric intuition behind dual numbers. I get the algebraic proofs. I understand that Dual number arithmetic is an application of the chain rule. But I was hoping that someone could explain some sort of graphical intuition behind what dual numbers are and why they can solve for derivatives and jacobians. Is there some graphical explanation for what a number represented by a value, and derivative are? How do these compose graphically to produce the overall derivative of the function?

Any help is appreciated.

  • 2
    $\begingroup$ I always thought of it as an algebraic way of representing a taylor series. The $\epsilon$ parameter represents a small change in value, but instead of $\epsilon^2$ just being small, we force $\epsilon^2=0$ identically. The derivative is simply computed by composition with the coefficient of the $\epsilon$ term yielding the derivative after expansion and collection of terms. If there is additional geometric intuition / interpretation beyond that, I’d love to know as well. $\endgroup$
    – Paul
    Jul 8 at 20:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.