Taylor expansion round-off error

Question

In the Numerical Recipes in section 5.7.- Numerical derivatives it's introduced de roundoff error of:

$$ f^{\prime}(x) \approx \frac{f(x+h)-f(x)}{h} $$

as (with $h$ an "exact" number):

$$ \tag{1} e_{r} \sim \epsilon_{f} \mid f(x) / h| $$

With the fractional accuracy comparable to the machine accuracy $\epsilon_{f} \approx \epsilon_{m}$.

Question: Where does the roundoff error expression $(1)$ come from?

Question: Let's say we have the third degree Taylor expansion:

$$ f(x+h) \approx f(x)+h f^{\prime}(x)+\frac{1}{2} h^{2} f^{\prime \prime}(x)+\frac{1}{6} h^{3} f^{\prime \prime \prime}(x) $$

What is the roundoff error of this Taylor expansion?

Answer 1

Let $g(x)=\frac{f(x+h)-f(x)}{h}$, and let $\bar{g}(x)$ its floating point representation with machine precision denoted by $\mu$. Recall that $\text{fl}(f(x)) = f(x)(1+\delta)$ with $|\delta| \leq \mu$.

We have $$|g(x)- \bar{g}(x)| = |\frac{f(x+h)-f(x) - (f(x+h)(1+\delta) - f(x) (1+\delta)}{h})| \leq \frac{2 \mu}{h}$$ where the last inequality follows from $|\delta| \leq \mu$

---

Notice that the error you'll observe is the following: $|f'(x) - \bar{g}(x)|$ i.e. the difference between the correct result and the representation in finite precision. We can estimate this with a simple trick:

$$|f'(x) - \bar{g}(x)|= |f'(x) - g(x) + g(x) - \bar{g}(x)| \leq C h + \frac{2 \mu}{h}$$ In the last step I used the triangle inequality + the F.D. truncation error (where $C$ of course depends on 2nd derivative of $f$ in the interval ) and the estimate above. In this way you can also compute the optimal $h$, i.e. $\bar{h} = \sqrt{\frac{2 \mu}{C}}$. As a consequence, you'll observe that for $0<h<\bar{h}$ the roundoff error will dominate and you'll keep loosing accuracy