Questions tagged [automatic-differentiation]

Often referred to as Algorithmic-Differentiation or AD -- a technique to automatically generate code that evaluates the derivative of a function. AD repeatedly applies the chain rule and classical rules of calculating derivatives. AD usually takes a block of code representing a function and returns a block of code representing that function's derivative.

Filter by
Sorted by
Tagged with
2 votes
1 answer
103 views

"Don't take the derivative of the approximation but approximate the derivative"..or something like this

Don't take the derivative of the approximation but approximate the derivative or something similar. I don't quite remember where I heard this but I am trying to find some work on the support or ...
Kapil's user avatar
  • 21
2 votes
2 answers
108 views

Automatic Differentiation In the Presence of Jump Points

I have a complex monte-carlo cashflow model that traditionally uses the finite difference (FD) method to calculate its derivative at any given point. To improve model performance, I coded forward-mode ...
Mild_Thornberry's user avatar
1 vote
1 answer
92 views

Automatic differentiation (AD) of a loss function which maps unitary matrix onto number

Is it possible to estimate whether automatic differentiation (AD) techniques could enable a more efficient way to repeatedly compute the derivative $\delta L / \delta u^*_{ij}$ of a specific loss ...
thyme's user avatar
  • 111
0 votes
1 answer
261 views

Computing the second derivative using Automatic Differentiation

Does anyone have any resources I could follow that explains how to compute the Nth derivative using both forward and backward autodiff I understand how to compute the first derivatives Any assistance ...
Gideon Ilung's user avatar
1 vote
1 answer
97 views

Automatic Differentiation using foward mode on matrices

Whilst googling I see reverse mode automatic differentiation (AD) tends to be used when optimising neural networks. Would it not be better to use forward mode and treat your input as a single variable,...
Gideon Ilung's user avatar
6 votes
1 answer
298 views

state of automatic differentiation

I've been working with TensorFlow and I'm very impressed with its automatic differentiation capabilities. I'm wondering what the state of the art in automatic differentiation for finite element ...
NNN's user avatar
  • 670
3 votes
2 answers
600 views

How to compute the Hessian using auto differentiation?

Assume that $f$ is defined as a composition of functions: $$f=f_2 \circ f_1$$ where $f_1:\mathbb{R}^n \rightarrow \mathbb{R}^{m_1}$ and $f_2:\mathbb{R}^{m_1} \rightarrow \mathbb{R}^n$. We can compute ...
user572780's user avatar
3 votes
2 answers
389 views

Automatic differentiation of a numerical solver

We often want to use numerical methods to evolve a system in time. That is, for a set of differential equations, we can specify some parameters $\bar{\theta}$ and pass these into our numerical solver ...
user1887919's user avatar
0 votes
0 answers
79 views

Automatic differentiation necessary for large optimal control problems?

I am investigating ways to solve an optimal control problem in an embedded way, preferably in Java. The system is modeled with triple integrator dynamics $u=\dddot{x}$ and solved with multiple ...
SAVG8778's user avatar
4 votes
3 answers
705 views

Why are dual numbers needed only in forward-mode autodiff?

I'm trying to understand autodiff better, and specifically the connection between autodiff and dual numbers, and why dual numbers are needed in the first place. The pytorch help pages about autodiff [...
Maverick Meerkat's user avatar
0 votes
0 answers
75 views

Automatic differentiation of integral

I would like to compute the integral $$ \frac{\partial}{\partial{x}} \int_{-1}^1 \int_{-1}^1 F(x,y,\xi, \eta) \; d\xi \; d\eta $$ or moving the derivative inside the integrals $$ \int_{-1}^1 \int_{-1}^...
Olumide's user avatar
  • 265
8 votes
0 answers
174 views

Is there a graphical interpretation or explanation of automatic differentiation compared to numerical differentiation

I have been looking at automatic differentiation for solving differential equations lately. I understand the basic ideas of using Dual numbers and such for finding derivatives, etc. However, I feel ...
krishnab's user avatar
  • 297
-2 votes
1 answer
40 views

Handling variables multiple times in Reverse-Mode Automatic Differentiation

If I try to derive a function computationally with Reverse-Mode Auto-Diff, I can derive a single function wrt. many variables in a single go. My Issue is now, what happens, if I input this function (...
Clebo Sevic's user avatar
5 votes
3 answers
238 views

Automatic finite differences

Given numbers $x, y \in \mathbb{R}$ where $$\frac{|y-x|}{|x|}$$ is small, and code that implements the function $f$ with a sequence of arithmetic operations, I would like to compute to high accuracy ...
Federico Poloni's user avatar
0 votes
0 answers
80 views

Uses for Automatic Differentiation in Investment Banking

I've finally wrapped my head around the advantages and disadvantages of AD/AAD compared to FD and SD, but could someone please explain to me where you'd use AD/AAD in an investment bank/banking ...
12Quant34's user avatar
0 votes
1 answer
144 views

How can the Lipschitz continuity be shown with power iteration method?

I know that the definition of the Lipschitz continuity is defines as $$||f(y) - f(x)|| \leq L ||y-x||$$ My professor told me that by knowing $f$ we can find constant $L$ using the power iteration ...
Karina's user avatar
  • 1
5 votes
0 answers
94 views

Can automatic differentiation be used on the parameters of an optimization problem?

If I wanted to perform an optimization using a Newton-based solver where the Hessian and gradient of a function are known analytically, and then use a package such as Adept to compute a Jacobian ...
Sal L's user avatar
  • 51
2 votes
1 answer
228 views

Rootfinding algorithm that takes advantage of automatic differentiation

Is there any algorithm (or tricks) for rootfinding to take advantages of automatic differentiation (AD)? Rootfinding algorithms typically solve $$ \mathbf{f}(\mathbf{y}) = \mathbf{0} $$ where $\mathbf{...
Firman's user avatar
  • 181
8 votes
1 answer
2k views

When and when not to use automatic differentiation

I am just learning (more) about automatic differentiation (AD) and at this stage it kind of seems like black magic to me. The second paragraph of its Wikipedia article makes it sound too good to be ...
tmph's user avatar
  • 337
1 vote
2 answers
1k views

Jacobians with automatic differentiation

I have an objective function F: Nx1 -> Nx1, where N>30000. There are many sparse matrix/tensor multiplications in this function, so taking an analytic Jacobian by paper and pen is cumbersome. ...
Someone's user avatar
  • 21
2 votes
2 answers
144 views

Methods for solving discrete PDEs using algorithmic differentiation results

I'm looking for a method to solve a 20000 variable, 20000 residual non-linear PDE with a Galerkin method. I have Fortran subroutines for: The residuals: $\vec{r}(\vec{x})$; Their Jacobian multiplied ...
Pedro Secchi's user avatar
1 vote
1 answer
115 views

Using adolc for the sign function in c++

Here is an implementation of the sign function in C++ using Adolc librairy for automatic differentiation. ...
Smilia's user avatar
  • 468
13 votes
4 answers
1k views

Example where autodiff works but symbolic differentiation will not?

According to the survey paper on autodiff (linked) Autodiff works on inputs that cannot be specified in closed form but can be described by a sequence of code, each component of which is ...
Lucas Roberts's user avatar
2 votes
1 answer
134 views

Truncated power series algebra implementation

1) I am looking for references for an efficient implementation and usage of TPSA. What sources exist besides Berz's 1989 original paper and the incomplete chapter in Dragt's book? 2) Are there ...
homocomputeris's user avatar
8 votes
1 answer
2k views

How fast is automatic differentiation?

I asked this question earlier on StackOverflow, but it's obviously better suited for SciComp: While there seem to be lots of references online which compare automatic differentiation methods and ...
gilgamec's user avatar
  • 181
2 votes
0 answers
81 views

Reverse automatic differentiation and integration

In Symplectic Runge-Kutta schemes for adjoint equations, automatic differentiation, optimal control and more Sanz Serna writes: It is well known that the reverse mode of differentiation implies ...
homocomputeris's user avatar
2 votes
1 answer
174 views

Automatic differentiation via ADOL-C and the Heaviside Function

I am writting a c++ program in which I define a function $$\displaystyle F(t) = \sum_{i}r_i\,H(t-t_i)$$ where $H$ is the heaviside function, $t_i$ are optimal parameters which are mutable. The ...
Smilia's user avatar
  • 468
14 votes
1 answer
2k views

Fast Automatic Differentiation for numpy?

I would like to use automatic differentiation to calculate gradients to function written in numpy. I've come across a number of packages, including autograd tangent chainer But none of them seem ...
user357269's user avatar
7 votes
1 answer
167 views

Automatic differentiation of barycentric rational functions

By a barycentric rational interpolant we understand a function of the form \begin{align*} r(t) := \frac{\sum_{i=0}^{n-1} \frac{w_i y_i}{t-t_i} }{ \sum_{i=0}^{n-1} \frac{w_i}{t-t_i}} \end{align*} In ...
user14717's user avatar
  • 2,155
2 votes
2 answers
722 views

Automatic Differentiation - reverse accumulation of linear system solve

I am studying the reverse mode of automatic differentiation. The reverse mode of automatic differentiation allows the efficient computation of a the derivative of a single dependent variable $y$ with ...
benno's user avatar
  • 301
0 votes
1 answer
3k views

Derivatives of a Chebychev polynomial

I am using Chebychev collocation nodes for approximation, and my problem requires me to calculate derivatives of the polynomial. I have been reading from a few sources, but I am not sure I understand ...
Anup Mulay's user avatar
2 votes
0 answers
272 views

Iterative linear solvers compatible with automatic differentiation?

I'm using automatic differentiation on a function that contains a sparse nonsymmetric linear system to be solved. I was using BiCGStab to solve this part of the function, but noticed the derivatives ...
bfletch's user avatar
  • 163
1 vote
0 answers
171 views

Is casadi suitable for data fitting?

Quite often I do fit some ODE or DAE systems to my data (small to medium sized problems). Via the assimulo package, I found Casadi and read a bit about the language modellica. Casadi offers automatic ...
Moritz's user avatar
  • 343
2 votes
1 answer
88 views

$(1+x^M)^{1/M}$ need to be able to calculate any order derivatives vs. $x$ and $M$ for $x\ge 0$ and $M\gt 2$

cannot delete my own question, so I try to overwrite it instead...
user avatar
2 votes
4 answers
770 views

Constructing sparsity pattern of the Jacobian of a FORTRAN subroutine

I need to calculate the Jacobian matrix of a subroutine F(U). Both F and U are of size N(=O($10^5$)). Using Tapenade, I differentiated the routine in tangent mode. I cannot calculate the full Jacobian ...
0b1100001's user avatar
  • 531
5 votes
6 answers
1k views

Open source auto-differentiation for MATLAB?

Are there any open-source auto-differentiation libraries for MATLAB? I am aware of commercial packages such as Tomlab/MAD and plenty of C++ libraries, but I can't find many more for MATLAB other than ...
jlperla's user avatar
  • 376
14 votes
2 answers
606 views

Regarding automatic differentiation, is source-code-transformation (STC) more efficient than operator-overloading (OO)?

We are working on a Bayesian model for a space-time process, and are using a No-U-Turn sampler (NUTS) that requires a model for the log-probability and it's gradient with respect to the model ...
Matthew Emmett's user avatar
1 vote
1 answer
2k views

Calculating Divergence in COMSOL

Is it computationally safe and accurate to use the following equation in COMSOL to compute the divergence of the vector quantity J (instead of using its general built-in equations that have $\nabla$ ...
Ali Abbasinasab's user avatar
25 votes
4 answers
3k views

When should I use C++ expression templates in computational science, and when should I *not* use them?

Suppose that I'm working on a scientific code in C++. In a recent discussion with a colleague, it was argued that expression templates could be a really bad thing, potentially making software ...
Geoff Oxberry's user avatar
14 votes
1 answer
386 views

Is there a tool out there that can generate interval extensions of Fortran (or C) functions by parsing Fortran (or C) code?

Case studies in my PhD thesis require that I have interval extensions of Fortran subroutines in CHEMKIN-II (apologies for the link; it's the best one I could find for a package no longer distributed ...
Geoff Oxberry's user avatar
16 votes
2 answers
1k views

When is automatic differentiation cheap?

Automatic differentiation allows us to numerically evaluate the derivative of a program on a particular input. There is a theorem that this computation can done at a cost less than five times the cost ...
MRocklin's user avatar
  • 3,088