As we know, machine epsilon limits relative rounding error in the range of normalized floating point numbers. But it is easy to check that this is not true for denormalized numbers.
My question is when one should take this subtlety into account and what sort of problems can this issue cause? Any references?
Back in the era of Intel 387 math coprocessors I had to maintain an interrupt handler for floating point exceptions. Apart from that, I agree that pretty much everyone ignores denormals (or subnormals as they are sometimes called), and this is successful in part because the IEEE 754 standard default handling of "gradual underflow" hides their (presumably rare) occurrance (better, at least, than overflow exceptions).
Denormals are an edge case of loss of significance, from exponent underflow. As a matter of theoretical treatment in numerical methods, it is most likely to arise by subtractive cancellation. But we will often have subtractive cancellation (whether benign or harmful) without creating denormals.
Some studies have been done on the slower performance caused by operations on denormals, such as Dooley and Kale (2013), Quantifying the Interference Caused by Subnormal Floating-point Values, and by the same authors with additional coauthors, Performance Degradation in the Presence of Subnormal Floating-Point Values.
Here's a blog article from 2012 that links back to a highly upvoted StackOverflow thread on the performance hit from denormalized floats.
Quantifying the effects on numerical accuracy are apt to be specific to the application, and in many cases the luxury of doing a comparison with multiple or arbitrary precision arithmetic may be the only way of getting a definitive measure.
It's worth a bit of a discussion as to why denormals don't matter in typical scientific computation. Usually, we are given some input $x$ and want to compute a function $f(x)$. Due to numerical errors, we instead compute $\tilde{f}(x) \approx f(x)$. There are two kinds of accuracy typically sought:
Denormals break relative, but usually do not break absolute. However, in most situations absolute accuracy is sufficient, and the only reason to establish relative accuracy is that it doesn't require knowledge of $\beta$ (this helps code and analysis modularity). Since denormals interfere with absolute accuracy typically only for $\beta$ much smaller than we ever need, all is well.
This is not always true: the canonical example is normalizing a vector where the output must have unit length to machine precision. Here relative accuracy really is what is required, since the magnitudes will be amplified up to detectable levels. In this case, denormals should be treated as equivalent to zero.
I've been doing numerical methods for 15 years and had to read up on what denormals actually are. Given that I converse a lot with others who are also doing numerical methods and had never heard anyone talk about it, it's safe to assume that everyone just ignores this oddity -- denormals play no role in everyday numerics.
