Questions tagged [automatic-differentiation]
Often referred to as Algorithmic-Differentiation or AD -- a technique to automatically generate code that evaluates the derivative of a function. AD repeatedly applies the chain rule and classical rules of calculating derivatives. AD usually takes a block of code representing a function and returns a block of code representing that function's derivative.
42 questions
0
votes
1
answer
205
views
Auto differentiation with JAX in python and ForwardDiff.jl in Julia give matrices with different values but same structure. Are the results right?
Using ForwardDiff in Julia gives me the following values for 2 matrices
I have tried to use JAX to auto differentiate the same function in Python but I am getting
the following results for A and B ...
- 11
2
votes
1
answer
126
views
"Don't take the derivative of the approximation but approximate the derivative"..or something like this
Don't take the derivative of the approximation but approximate the derivative
or something similar.
I don't quite remember where I heard this but I am trying to find some work on the support or ...
- 21
2
votes
2
answers
133
views
Automatic Differentiation In the Presence of Jump Points
I have a complex monte-carlo cashflow model that traditionally uses the finite difference (FD) method to calculate its derivative at any given point. To improve model performance, I coded forward-mode ...
- 123
1
vote
1
answer
103
views
Automatic differentiation (AD) of a loss function which maps unitary matrix onto number
Is it possible to estimate whether automatic differentiation (AD) techniques could enable a more efficient way to repeatedly compute the derivative $\delta L / \delta u^*_{ij}$ of a specific loss ...
- 111
0
votes
1
answer
482
views
Computing the second derivative using Automatic Differentiation
Does anyone have any resources I could follow that explains how to compute the Nth derivative using both forward and backward autodiff
I understand how to compute the first derivatives
Any assistance ...
- 13
1
vote
1
answer
106
views
Automatic Differentiation using foward mode on matrices
Whilst googling I see reverse mode automatic differentiation (AD) tends to be used when optimising neural networks.
Would it not be better to use forward mode and treat your input as a single variable,...
- 13
7
votes
1
answer
329
views
state of automatic differentiation
I've been working with TensorFlow and I'm very impressed with its automatic differentiation capabilities. I'm wondering what the state of the art in automatic differentiation for finite element ...
- 852
3
votes
2
answers
993
views
How to compute the Hessian using auto differentiation?
Assume that $f$ is defined as a composition of functions:
$$f=f_2 \circ f_1$$
where $f_1:\mathbb{R}^n \rightarrow \mathbb{R}^{m_1}$ and $f_2:\mathbb{R}^{m_1} \rightarrow \mathbb{R}^n$. We can compute ...
- 169
3
votes
2
answers
611
views
Automatic differentiation of a numerical solver
We often want to use numerical methods to evolve a system in time.
That is, for a set of differential equations, we can specify some parameters $\bar{\theta}$ and pass these into our numerical solver ...
- 233
0
votes
0
answers
90
views
Automatic differentiation necessary for large optimal control problems?
I am investigating ways to solve an optimal control problem in an embedded way, preferably in Java. The system is modeled with triple integrator dynamics $u=\dddot{x}$ and solved with multiple ...
- 1
4
votes
3
answers
1k
views
Why are dual numbers needed only in forward-mode autodiff?
I'm trying to understand autodiff better, and specifically the connection between autodiff and dual numbers, and why dual numbers are needed in the first place.
The pytorch help pages about autodiff [...
- 157
0
votes
0
answers
79
views
Automatic differentiation of integral
I would like to compute the integral
$$
\frac{\partial}{\partial{x}} \int_{-1}^1 \int_{-1}^1 F(x,y,\xi, \eta) \; d\xi \; d\eta
$$
or moving the derivative inside the integrals
$$
\int_{-1}^1 \int_{-1}^...
- 325
8
votes
0
answers
177
views
Is there a graphical interpretation or explanation of automatic differentiation compared to numerical differentiation
I have been looking at automatic differentiation for solving differential equations lately. I understand the basic ideas of using Dual numbers and such for finding derivatives, etc. However, I feel ...
- 297
-2
votes
1
answer
41
views
Handling variables multiple times in Reverse-Mode Automatic Differentiation
If I try to derive a function computationally with Reverse-Mode Auto-Diff, I can derive a single function wrt. many variables in a single go.
My Issue is now, what happens, if I input this function (...
- 97
7
votes
4
answers
327
views
Automatic finite differences
Given numbers $x, y \in \mathbb{R}$ where
$$\frac{|y-x|}{|x|}$$
is small, and code that implements the function $f$ with a sequence of arithmetic operations, I would like to compute to high accuracy ...
- 12.6k
0
votes
0
answers
81
views
Uses for Automatic Differentiation in Investment Banking
I've finally wrapped my head around the advantages and disadvantages of AD/AAD compared to FD and SD, but could someone please explain to me where you'd use AD/AAD in an investment bank/banking ...
- 17
0
votes
1
answer
175
views
How can the Lipschitz continuity be shown with power iteration method?
I know that the definition of the Lipschitz continuity is defines as $$||f(y) - f(x)|| \leq L ||y-x||$$
My professor told me that by knowing $f$ we can find constant $L$ using the power iteration ...
- 1
5
votes
0
answers
96
views
Can automatic differentiation be used on the parameters of an optimization problem?
If I wanted to perform an optimization using a Newton-based solver where the Hessian and gradient of a function are known analytically, and then use a package such as Adept to compute a Jacobian ...
- 51
2
votes
1
answer
241
views
Rootfinding algorithm that takes advantage of automatic differentiation
Is there any algorithm (or tricks) for rootfinding to take advantages of automatic differentiation (AD)?
Rootfinding algorithms typically solve
$$
\mathbf{f}(\mathbf{y}) = \mathbf{0}
$$
where $\mathbf{...
- 191
10
votes
1
answer
3k
views
When and when not to use automatic differentiation
I am just learning (more) about automatic differentiation (AD) and at this stage it kind of seems like black magic to me. The second paragraph of its Wikipedia article makes it sound too good to be ...
- 357
2
votes
2
answers
2k
views
Jacobians with automatic differentiation
I have an objective function F: Nx1 -> Nx1, where N>30000. There are many sparse matrix/tensor multiplications in this function, so taking an analytic Jacobian by paper and pen is cumbersome.
...
- 31
2
votes
2
answers
159
views
Methods for solving discrete PDEs using algorithmic differentiation results
I'm looking for a method to solve a 20000 variable, 20000 residual non-linear PDE with a Galerkin method.
I have Fortran subroutines for:
The residuals: $\vec{r}(\vec{x})$;
Their Jacobian multiplied ...
- 91
1
vote
1
answer
133
views
Using adolc for the sign function in c++
Here is an implementation of the sign function in C++ using Adolc librairy for automatic differentiation.
...
- 488
13
votes
4
answers
1k
views
Example where autodiff works but symbolic differentiation will not?
According to the survey paper on autodiff (linked) Autodiff works on inputs that cannot be specified in closed form but can be described by a sequence of code, each component of which is ...
- 231
2
votes
1
answer
142
views
Truncated power series algebra implementation
1) I am looking for references for an efficient implementation and usage of TPSA. What sources exist besides Berz's 1989 original paper and the incomplete chapter in Dragt's book?
2) Are there ...
- 417
9
votes
1
answer
2k
views
How fast is automatic differentiation?
I asked this question earlier on StackOverflow, but it's obviously better suited for SciComp:
While there seem to be lots of references online which compare automatic differentiation methods and ...
- 191
2
votes
0
answers
92
views
Reverse automatic differentiation and integration
In Symplectic Runge-Kutta schemes for adjoint equations, automatic differentiation, optimal control and more Sanz Serna writes:
It is well known that the reverse mode of differentiation implies
...
- 417
2
votes
1
answer
188
views
Automatic differentiation via ADOL-C and the Heaviside Function
I am writting a c++ program in which I define a function $$\displaystyle F(t) = \sum_{i}r_i\,H(t-t_i)$$ where $H$ is the heaviside function, $t_i$ are optimal parameters which are mutable.
The ...
- 488
15
votes
1
answer
2k
views
Fast Automatic Differentiation for numpy?
I would like to use automatic differentiation to calculate gradients to function written in numpy.
I've come across a number of packages, including
autograd
tangent
chainer
But none of them seem ...
- 373
7
votes
2
answers
221
views
Automatic differentiation of barycentric rational functions
By a barycentric rational interpolant we understand a function of the form
\begin{align*}
r(t) := \frac{\sum_{i=0}^{n-1} \frac{w_i y_i}{t-t_i} }{ \sum_{i=0}^{n-1} \frac{w_i}{t-t_i}}
\end{align*}
In ...
- 2,165
2
votes
2
answers
930
views
Automatic Differentiation - reverse accumulation of linear system solve
I am studying the reverse mode of
automatic differentiation.
The reverse mode of automatic differentiation allows the efficient computation of a the derivative of a single dependent variable $y$ with ...
- 301
0
votes
1
answer
4k
views
Derivatives of a Chebychev polynomial
I am using Chebychev collocation nodes for approximation, and my problem requires me to calculate derivatives of the polynomial. I have been reading from a few sources, but I am not sure I understand ...
2
votes
0
answers
281
views
Iterative linear solvers compatible with automatic differentiation?
I'm using automatic differentiation on a function that contains a sparse nonsymmetric linear system to be solved. I was using BiCGStab to solve this part of the function, but noticed the derivatives ...
- 173
1
vote
0
answers
187
views
Is casadi suitable for data fitting?
Quite often I do fit some ODE or DAE systems to my data (small to medium sized problems). Via the assimulo package, I found Casadi and read a bit about the language modellica.
Casadi offers automatic ...
- 343
2
votes
1
answer
89
views
$(1+x^M)^{1/M}$ need to be able to calculate any order derivatives vs. $x$ and $M$ for $x\ge 0$ and $M\gt 2$
cannot delete my own question, so I try to overwrite it instead...
user7098
2
votes
4
answers
803
views
Constructing sparsity pattern of the Jacobian of a FORTRAN subroutine
I need to calculate the Jacobian matrix of a subroutine F(U). Both F and U are of size N(=O($10^5$)). Using Tapenade, I differentiated the routine in tangent mode. I cannot calculate the full Jacobian ...
- 531
5
votes
6
answers
1k
views
Open source auto-differentiation for MATLAB?
Are there any open-source auto-differentiation libraries for MATLAB?
I am aware of commercial packages such as Tomlab/MAD and plenty of C++ libraries, but I can't find many more for MATLAB other than ...
- 376
14
votes
2
answers
639
views
Regarding automatic differentiation, is source-code-transformation (STC) more efficient than operator-overloading (OO)?
We are working on a Bayesian model for a space-time process, and are using a No-U-Turn sampler (NUTS) that requires a model for the log-probability and it's gradient with respect to the model ...
- 2,086
1
vote
1
answer
2k
views
Calculating Divergence in COMSOL
Is it computationally safe and accurate to use the following equation in COMSOL to compute the divergence of the vector quantity J (instead of using its general built-in equations that have $\nabla$ ...
- 183
25
votes
4
answers
3k
views
When should I use C++ expression templates in computational science, and when should I *not* use them?
Suppose that I'm working on a scientific code in C++. In a recent discussion with a colleague, it was argued that expression templates could be a really bad thing, potentially making software ...
- 30.4k
14
votes
1
answer
392
views
Is there a tool out there that can generate interval extensions of Fortran (or C) functions by parsing Fortran (or C) code?
Case studies in my PhD thesis require that I have interval extensions of Fortran subroutines in CHEMKIN-II (apologies for the link; it's the best one I could find for a package no longer distributed ...
- 30.4k
17
votes
2
answers
1k
views
When is automatic differentiation cheap?
Automatic differentiation allows us to numerically evaluate the derivative of a program on a particular input. There is a theorem that this computation can done at a cost less than five times the cost ...
- 3,098