We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [floating-point]

A method of representing numbers by a fixed number of significant digits, and the exponent of some base number. They are characterized in the form ${(significant digits)}*base^{exponent}$. Typically, numbers are represented with respect to base = 2 (binary).

12 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
8
votes
0answers
473 views

Implementing std::nextafter: Should denormals-are-zero mode affect it? If so, how?

This might be the wrong stackexchange site for this question. math.SE, cs.SE, programmers.SE, and of course stackoverflow are all possibilities. I'm hoping to reach an audience that might actually ...
5
votes
0answers
47 views

Evaluate Nth root of a rational to a correctly rounded float

Excuse my lack of vocabulary for I have no formal training in this field, which is also why I ask this question - it may be trivial or it may be impossible. I want to evaluate an expression in the ...
5
votes
0answers
186 views

Any way to avoid catastrophic cancellation when computing the discriminant of a quadratic function?

Homework disclaimer... The task: We are using the following algorithm to solve the quadratic equation $x^2+2px+q=0$: $x_1=|p|+\sqrt{p^2-q}\mathtt{;}$ $\mathtt{if}\,p>0\,\mathtt{then}\,...
4
votes
0answers
150 views

Is there any catch on using `zgemm3m` vs regular `zgemm`?

I've just (to my embarrassment) encountered a BLAS-like extension of a matrix-matrix product subroutine gemm in Intel MKL: gemm3m...
4
votes
0answers
79 views

Comparing sum of floating points

I am currently working on a numerical algorithm involving a lot of floating point arithmetic, involving some badly conditioned problem sets. I am using the relation $|x - y| / (\max(|x|, |y|, 1)) \...
3
votes
0answers
84 views

Stabilizing online average calculation

In Knuth, the following method for computing an average is presented: \begin{align*} M_{n} = M_{n-1} + (x_{n} - M_{n-1})/n \end{align*} (See here, if you don't have TAOCP.) Assuming the samples all ...
3
votes
0answers
74 views

What disables Gradual Overflow?

After studying the process of Gradual Underflow, I'm left a little curious as to why machines don't implement Gradual Overflow; where numbers exceeding the overflow level would be stored as un-...
3
votes
1answer
164 views

Rounding errors in images of Julia sets

One typically computes Julia sets by iterating a complex function, such as a polynomial or rational function. How do rounding errors affect the results? I'm looking for references on this issue, ...
2
votes
0answers
45 views

Quick evaluation of floating point Absolute Error

I need to to find a quick and dirty way to estimate the absolute error introduced by a series of agebraic operations of IEEE single precision floating point numbers, a pessimistic result is ok. The ...
1
vote
0answers
77 views

Why is the method of im2col with GEMM is more efficient than the method of direction implementation with SIMD in CNN

The convolutional layers are most computationally intense parts of Convolutional neural networks (CNNs).Currently the common approach to impement convolutional layers is to expand the image into a ...
1
vote
0answers
83 views

Kahan Summation for Three-Term Recurrences

Kahan summation applies to summation problems, but not to three-term recurrence relations. However, a three-term recurrence shares many of the features of a summation-albeit with a rescaling step at ...
1
vote
0answers
39 views

t digits are used to represent the mantissa in floating-point system, but rounding unit is calculated for doubles with 53 bits

I'm reading the book "A First Course in Numerical Methods" by U. Ascher and C. Greif, and in the 2nd chapter it's written that ... we associate $x$ a floating point representation $fl(x)$ of the ...