Questions tagged [precision]

Issues related to the representation of numerical quantities in a finite representation in a given base differing from their exact mathematical value.

82 questions
Filter by
Sorted by
Tagged with
143 views

Accuracy loss in single-precision Euclidean norm computation

I do hydrodynamics simulations with Fortran and recently I met with this issue: I have a single-precision array b of length ...
• 23
88 views

High precision numerical integration of discrete data with Matlab

I have discrete data of a function plotted below: The "Y" values of the function near "X=1.57" are very close to each other and zero, like 9.25558265263186E-11 and 5....
• 23
61 views

• 137
189 views

Numerically stable and fast sum of last K elements in sequence

Suppose I have a long, possibly infinite, sequence $x := [x_1, x_2, ...]$, and I want to use it to compute another sequence $y:=[y_1, y_2, ...]$ where each element is the sum of the last K elements of ...
• 171
177 views

Best way to check if SOR solution has converged for 2d matrix

I have written a SOR algorithm to solve the Laplace equation on a 2d grid. The outside of the grid is fixed at 0 and the central square is fixed at 10. I can obtain the fully converged solution for ...
335 views

log-sum-exp trick for signed/complex numbers

I need to evaluate a sum of values that are on many different orders of magnitude in scale but might be signed. I’ve had great luck with the “log-sum-exp” trick for an unsigned version of my problem, ...
• 1,331
122 views

How do I globally change the precision of a piece of code in Python to debug it?

I am solving a system of non-linear equations using the Newton-Raphson method in Python. This involves using the solve(Ax,b) function (...
359 views

Numerical stability in the product of many matrices

I have to calculate in numpy the matrix-product of many matrices (~400). Are there common practices to increase numerical stability? If this is relevant, the matrices are $300\times 300$ orthogonal ...
• 129
347 views

How to perform an eigendecomposition of a general complex matrix with arbitrary precision in C/C++

I need to obtain the Eigenvectors of a general complex matrix, but with quadruple precision. Is anyone aware of a means to do this? I currently use Tux Eigen, and I see that in their unsupported ...
• 131
859 views

• 173
1 vote
161 views

How to deal with big numbers in intermediate calculations?

I have a rather long expression (https://pastebin.com/jUsxdCCs) that is an analytical solution of a set of differential equations generated symbolically from Maple. I need to solve a set of equations ...
188 views

How to deal with very low numerical values in C?

I have to work with values such as 1e-15 in my code but I can't. Indeed, because of low precision these values are equivalent to 0. Any ideas?
521 views

Precision loss in Matrix-Vector product when applying Finite-Difference scheme

I am applying a 6th order Finite-Difference differentiation scheme as seen in http://www.scholarpedia.org/article/Method_of_lines/example_implementation/dss006 There seems to be severe numerical/...
1 vote
372 views

Any FOSS MATLAB/Octave toolbox for high-speed variable precision arithmetic?

I need to use variable precision arithmetic in MATLAB for an expensive set of computation. The vpa function provided by the symbolic math toolbox is very slow. I found a non-free alternative toolbox ...
238 views

What is the error associated with Fornberg's algorithm?

Bengt Fornberg derived a general way to compute the weights for arbitrary finite difference schemes in two papers: his 1988 paper and (better) his 1998 paper. What are the numerical errors ...
• 263
74 views

• 149