2
$\begingroup$

Don't take the derivative of the approximation but approximate the derivative

or something similar.

I don't quite remember where I heard this but I am trying to find some work on the support or denial of this statement especially when using AD to approximate (calculate derivative) of a function approximation (say a neural network). I am trying to do a little experiment on why we need two neural nets (one for the function itself and one for derivative) instead of using autodiff to get the derivative of the approximated function.

$\endgroup$
3
  • $\begingroup$ Please expand on how this derivative is then used. Is this just a replacement for the gradient for the learning algorithm, or, as example, learning a surface and the surface normal? That is, approximating the function in $C^1$, having a Jacobian matrix as part of the given output in the training data? $\endgroup$ Dec 8, 2023 at 8:40
  • $\begingroup$ @LutzLehmann It will be used get the gradient and chain it with other gradients to get the total derivative of a function which is then passed to an optimizer $\endgroup$
    – Kapil
    Dec 8, 2023 at 16:49
  • $\begingroup$ For the widely-used class of polynomial approximations, the two are actually identical (--if done consistently). Just mentioning, even if it's not in scope of the question. $\endgroup$
    – davidhigh
    Dec 9, 2023 at 8:25

1 Answer 1

1
$\begingroup$

It is heuristics.

There are 2 or 3 ways to look at it & we will get the necessary Conclusion.

(1) Differentiation is Exact & "undoable" ( within known limits ) & it is not losing information & not introducing Error.
Approximation is not "undoable" & it is losing information & introducing Error.
Hence we should try to keep maximum information & execute all calculations , including Differentiation , earlier & then use Approximation at the end to have minimum information loss & minimum Error.
When we try Approximation earlier , then further calculations , including Differentiation , will compound the information loss & multiply the Error.

(2) Consider Mathematical Example : $\sin(x)$ & we want Approximate values of the Derivative at $x=0.00001,0.0002,0.003,\cdots$
(2A) When we use Approximation $\sin(x) \approx x$ & then take Derivative , we will get Constant $1$ & we have to use that Constant at $x=0.00001,0.0002,0.003,\cdots$ to get Constant Curve.
(2B) When we take Derivative , we get $\cos(x)$ & then we can use Approximation , eg $\cos(x) \approx 1 - x^2/2$ & we can calculate that at $x=0.00001,0.0002,0.003,\cdots$ to get Non-Constant Curve.

In general , (2B) is better than (2A) practically.

(3) When we Approximate , we are making bounds on the values. When we then take Derivative , we will still have some bounds.
When we Differentiate , we might get unbounded values or larger bounds for the values. When we then take Approximations , we will still have the larger range.

Conclusion : In general , the Statement by OP is valid.

ADDENDUM :

WordWeb gives this :

heuristic :

  • Of or relating to or using a general formulation that serves to guide investigation
  • A commonsense rule (or set of rules) intended to increase the probability of solving some problem

In other words , Heuristic thinking is not in terms of "true or false" , it is in terms of "valid or invalid" & "useful or not useful" , in general.

The Heuristic Statement by OP is valid & useful in general.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.