# Questions tagged [precision]

Issues related to the representation of numerical quantities in a finite representation in a given base differing from their exact mathematical value.

90 questions
Filter by
Sorted by
Tagged with
114 views

### Single precision vs double precision conjugate gradients

I tested my conjugate gradients implementation with float and double precision and contrary to my guess the double code was twice faster than the single precision code. The reason is that I need many ...
90 views

### Runge Kutta 4th order: unexpected result

My problem in brief: in some situations, the Runge Kutta 4th order method (RK4) doesn't seem to give 4th order improvement when using a smaller time step. I wonder how this worse-than-expected result ...
1 vote
227 views

### Float equality tolerance for single and half precision

Suppose the metric is abs(a-b) <= rtol * max(abs(a), abs(b)) i.e. math.isclose with ...
170 views

### Numerically stable way to implement Cramer's rule analog

Problem statement Let $A$ be an $n\times n$ matrix and $b$ an $n$-dimensional vector. For $j\in \{1, \dots, n \}$, let $A_j$ be the matrix where we take $A$ and replace the $j^{\rm th}$ column with $b$...
68 views

### Dynamic tolerance in a conditional loop to obtain maximum precision allowed by machine floating point numbers

I have coded a simple program for a root finding problem using Halley's method. Here is the code: ...
116 views

### Summation of trigonometric functions results in error with finite precision

Consider the following expression: $$f(t) = B+\sum_{k=1}^{N} A_k\cos(\omega_kt)$$ where $A$ and $B$ are known. the frequencies are also known but are not multiples of a fundamental frequency. However, ...
112 views

1 vote
171 views

### How to deal with big numbers in intermediate calculations?

I have a rather long expression (https://pastebin.com/jUsxdCCs) that is an analytical solution of a set of differential equations generated symbolically from Maple. I need to solve a set of equations ...
201 views

### How to deal with very low numerical values in C?

I have to work with values such as 1e-15 in my code but I can't. Indeed, because of low precision these values are equivalent to 0. Any ideas?
558 views

### Precision loss in Matrix-Vector product when applying Finite-Difference scheme

I am applying a 6th order Finite-Difference differentiation scheme as seen in http://www.scholarpedia.org/article/Method_of_lines/example_implementation/dss006 There seems to be severe numerical/...
1 vote