"Don't take the derivative of the approximation but approximate the derivative"..or something like this

Don't take the derivative of the approximation but approximate the derivative

or something similar.

I don't quite remember where I heard this but I am trying to find some work on the support or denial of this statement especially when using AD to approximate (calculate derivative) of a function approximation (say a neural network). I am trying to do a little experiment on why we need two neural nets (one for the function itself and one for derivative) instead of using autodiff to get the derivative of the approximated function.

• Please expand on how this derivative is then used. Is this just a replacement for the gradient for the learning algorithm, or, as example, learning a surface and the surface normal? That is, approximating the function in $C^1$, having a Jacobian matrix as part of the given output in the training data? Dec 8, 2023 at 8:40
• @LutzLehmann It will be used get the gradient and chain it with other gradients to get the total derivative of a function which is then passed to an optimizer Dec 8, 2023 at 16:49
• For the widely-used class of polynomial approximations, the two are actually identical (--if done consistently). Just mentioning, even if it's not in scope of the question. Dec 9, 2023 at 8:25

It is heuristics.

There are 2 or 3 ways to look at it & we will get the necessary Conclusion.

(1) Differentiation is Exact & "undoable" ( within known limits ) & it is not losing information & not introducing Error.
Approximation is not "undoable" & it is losing information & introducing Error.
Hence we should try to keep maximum information & execute all calculations , including Differentiation , earlier & then use Approximation at the end to have minimum information loss & minimum Error.
When we try Approximation earlier , then further calculations , including Differentiation , will compound the information loss & multiply the Error.

(2) Consider Mathematical Example : $$\sin(x)$$ & we want Approximate values of the Derivative at $$x=0.00001,0.0002,0.003,\cdots$$
(2A) When we use Approximation $$\sin(x) \approx x$$ & then take Derivative , we will get Constant $$1$$ & we have to use that Constant at $$x=0.00001,0.0002,0.003,\cdots$$ to get Constant Curve.
(2B) When we take Derivative , we get $$\cos(x)$$ & then we can use Approximation , eg $$\cos(x) \approx 1 - x^2/2$$ & we can calculate that at $$x=0.00001,0.0002,0.003,\cdots$$ to get Non-Constant Curve.

In general , (2B) is better than (2A) practically.

(3) When we Approximate , we are making bounds on the values. When we then take Derivative , we will still have some bounds.
When we Differentiate , we might get unbounded values or larger bounds for the values. When we then take Approximations , we will still have the larger range.

Conclusion : In general , the Statement by OP is valid.