# Questions tagged [precision]

Issues related to the representation of numerical quantities in a finite representation in a given base differing from their exact mathematical value.

92 questions
Filter by
Sorted by
Tagged with
79 views

### Computing $\frac{x - y}{x - z}$ when $x,y,z$ are close to each other

What is the most stable way to compute $$\frac{x - y}{x - z}$$ when $x$, $y$, and $z$ are all close to each other? I would like to compute expressions of this form in low precision on a GPU, but when ...
• 3,195
197 views

### Unexpected result when summing sorted (and unsorted) positive floating point numbers

I am exploring Higham's excellent Accuracy and Stability of Numerical Algorithms and chapter 4 is dedicated to summation. So I decided to test the most basic thing. Summing positive random numbers ...
• 71
141 views

### Single precision vs double precision conjugate gradients

I tested my conjugate gradients implementation with float and double precision and contrary to my guess the double code was twice faster than the single precision code. The reason is that I need many ...
• 2,352
106 views

### Runge Kutta 4th order: unexpected result

My problem in brief: in some situations, the Runge Kutta 4th order method (RK4) doesn't seem to give 4th order improvement when using a smaller time step. I wonder how this worse-than-expected result ...
• 121
1 vote
827 views

### Float equality tolerance for single and half precision

Suppose the metric is abs(a-b) <= rtol * max(abs(a), abs(b)) i.e. math.isclose with ...
181 views

### Numerically stable way to implement Cramer's rule analog

Problem statement Let $A$ be an $n\times n$ matrix and $b$ an $n$-dimensional vector. For $j\in \{1, \dots, n \}$, let $A_j$ be the matrix where we take $A$ and replace the $j^{\rm th}$ column with $b$...
• 31
80 views

### Dynamic tolerance in a conditional loop to obtain maximum precision allowed by machine floating point numbers

I have coded a simple program for a root finding problem using Halley's method. Here is the code: ...
125 views

### Summation of trigonometric functions results in error with finite precision

Consider the following expression: $$f(t) = B+\sum_{k=1}^{N} A_k\cos(\omega_kt)$$ where $A$ and $B$ are known. the frequencies are also known but are not multiples of a fundamental frequency. However, ...
125 views

• 183
1 vote
173 views

### How to deal with big numbers in intermediate calculations?

I have a rather long expression (https://pastebin.com/jUsxdCCs) that is an analytical solution of a set of differential equations generated symbolically from Maple. I need to solve a set of equations ...