# How to derive the adjoint sensitivity equations for a least squares objective function gradient

The Problem

I would like to determine the gradient of a least squares objective function which depends on a vector of 40 parameters $$p$$, and the solution of a system of 32 differential equations. In specific, let the aforementioned system be:

$$\dot{y} = f(t,y(t,p),p) \tag{1}$$

then the least squares objective function is

$$G(t,p) = \frac{1}{2}\sum^n_{i=1}||y(t_i,p) - \hat{y}(t_i)||^2\tag{2}$$

where $$\hat{y}$$ is some measured data, and $$||\cdot||$$ is the L-2 norm.

My Attempt

For a small number of parameters I can use the typical approach employing the sensitivity equations which yields the following for the gradient of the objective function:

$$\frac{d}{dp}G(t,p) = \sum^n_{i=1}\left(y(t_i,p) - \hat{y}(t_i)\right)\frac{\partial y(t_i,p)}{\partial p} \tag{3}$$

where $$\frac{\partial y(t,p)}{\partial p}$$ is determined by solving the sensitivity equations simultaneously with $$(1)$$, which is one extra system of 32 differential equations for every value of $$p$$. For a small number of parameters $$p$$ the equation systems are kept reasonably small. For my case however with 40 parameters, each with a system of 32 equations to solve, there is a total of 1312 to solve which makes it very demanding on the solver.

I think that the solution for when the sensitivities of many parameters need to be calculated is to use the adjoint sensitivity equations where according to this website, they are as follows:

For an objective function $$G(t,p) = \int^T_0 g(t,p)dt \tag{5}$$ we have to solve a single extra differential equation for $$\lambda^*$$, namely: $$\frac{d\lambda^*}{dt} = \frac{\partial g}{\partial y} - \lambda^* J_y, \ \lambda^* = 0 \tag{6}$$ Then the gradient of the objective function is $$\frac{d}{dp}G(t,p) = -\int_{0}^{T}\lambda^{\star}\frac{\partial f}{\partial p}dt-\lambda^{T}(0)\frac{\partial y}{\partial p} \tag{7}$$

The Question

1. What is the function $$g$$ for my case where I have the least squares objective function?
2. How do I determine the value of $$\frac{\partial y}{\partial p}$$? Isn't the point of this approach to avoid the calculation of this value so that I don't have to use the sensitivity equations?
3. For the case of the least squares objective function in eqn. $$(2)$$, how is the gradient formulated using the adjoint method shown in equation $$(7)$$?

EDIT: After much searching I found the answer to the first question in this paper. Because equation $$(5)$$ is inheritly continuous, the least squares formulation has to be translated to such a format using the Diract Delta function $$\delta(t)$$: $$G(t,p) = \int^T_0g(t,p)dt=\int^T_0\sum^n_{i=1}\sum^n_{j=1}\frac{(y_j(t_i,p)-\hat{y}_j(t_i))^2}{2}\delta(t_i-t)dt \tag{8}$$ As a result equation $$(6)$$ becomes a discontinuous differential equation: $$\frac{d\lambda^*}{dt}=\sum^n_{i=1}\left(y(t_i,p) - \hat{y}(t_i)\right)^2\delta(t-t_i) -\lambda^*J_y \tag{9}$$

• I don't understand eq. (2). On the left, you have a sum over individual time points, but on the right you evaluate $y$ at a generic "$t$". Is your norm supposed to be the $L_2$ norm in time over the interval $[0,T]$? – Wolfgang Bangerth Nov 26 '20 at 17:10
• As you said, the norm is supposed to be evaluated for a vector $y(t,p)$, for all $t$. Having a second look at it, it doesn't add anything to the description of the problem, as its just a reformulation of the sum expression. I think I'll edit this one out. – kostas1335 Nov 26 '20 at 19:10
• Do you have the known values at $y$ at only a small number of points in time or can do you have them (or can easily interpolate) at essentially all points? – Brian Borchers Nov 26 '20 at 20:31
• I can have the measurement vector $\hat{y}$ at all the points I desire, as its can be accurately interpolated, so as you said I can have them at 'essentially all points'. – kostas1335 Nov 26 '20 at 21:04