Skip to main content
33 votes

C, Julia, Python, Maxima, Mathematica, ChatGPT and numerical errors

This function, as any similar hill, tent or hat functions, implements the expansion-and-folding scheme. This means that in the most benign interpretation you lose in each iteration one bit from the ...
Lutz Lehmann's user avatar
  • 6,109
31 votes

C, Julia, Python, Maxima, Mathematica, ChatGPT and numerical errors

I thought it would be interesting to see how the number of bits of precision affects the error. I wrote the following using arb, a library for interval arithmetic (and in particular a library that ...
davidlowryduda's user avatar
29 votes
Accepted

Is half precision supported by modern architecture?

Intel support for IEEE float16 storage format Intel supports IEEE half as a storage type in processors since Ivy Bridge (2013). Storage type means you can get a memory/cache capacity/bandwidth ...
Jeff Hammond's user avatar
  • 2,126
24 votes

Meaning of "-0.0" in Python?

Floating point numbers (according to the standard1 nearly all programming languages use) are stored with a certain number of bits in the mantissa, in the exponent, and with a sign bit. As such, ...
Wolfgang Bangerth's user avatar
23 votes
Accepted

Why are log and exp considered 'expensive' computations in ML?

To add to Lutz Lehmann's answer, you can look up the latency for the CPU instructions in this comprehensive table by Agner Fog. For example, on the Intel Ivy Bridge processors: FADD / FSUB (floating ...
Daniel Shapero's user avatar
20 votes
Accepted

What are some good strategies to test a floating point arithmetic implementation for double numbers?

You should test transition points. Floating-point numbers have several distinct "ranges": Standard/Normal arithmetic Subnormal arithmetic Infinite arithmetic NaN arithmetic Zero arithmetic ...
Richard's user avatar
  • 3,971
16 votes

C, Julia, Python, Maxima, Mathematica, ChatGPT and numerical errors

Other answers have explained the results from Python, Julia, Maxima, C, and Mathematica. I'll explain the result you got from ChatGPT. Generally, the way that ChatGPT works is that you ask a question, ...
Tanner Swett's user avatar
15 votes
Accepted

Are BLAS implementations guaranteed to give the exact same result?

No, that is not guaranteed. If you are using a NETLIB BLAS without any optimizations, it it mostly true that the results are the same. But for any practical usage of BLAS and LAPACK one uses a highly ...
M.K. aka Grisu's user avatar
15 votes

Matrix multiplication accuracy Matlab vs Python

First, see Mark L. Stone's answers, which is completely correct. Second, realize that this is the reason why people told you to use relative errors in your numerical analysis class. :) Third, the ...
Federico Poloni's user avatar
14 votes

Matrix multiplication accuracy Matlab vs Python

Here is R1, as computed in MATLAB: ...
Mark L. Stone's user avatar
13 votes

C, Julia, Python, Maxima, Mathematica, ChatGPT and numerical errors

Perhaps the reason you are stunned is that you don't understand that what you are asking to calculate is impossible to ever calculate exactly. So if you want the answer to some precision, you need to ...
Secto Kia's user avatar
  • 239
12 votes
Accepted

Evaluating $\log(\exp(x)+1)$ for negative $x$

Use the (IEEE standard) library function log1p, which should be present in all programming languages. The function log1p(x) ...
Federico Poloni's user avatar
12 votes

C, Julia, Python, Maxima, Mathematica, ChatGPT and numerical errors

Unfortunately this comes down to the math library. IEEE754 does not mandate exactly how the cosine should be computed (technically one doesn't even need to include a function to compute the cosine to ...
Federico Poloni's user avatar
12 votes

C, Julia, Python, Maxima, Mathematica, ChatGPT and numerical errors

We could use exact arithmetic to compute the correct value. A range of exact numbers $e$ correspond to the same $a=5.0$ representation in float32 IEEE754, plot over this range: You can see that after ...
Yaroslav Bulatov's user avatar
11 votes

Why are log and exp considered 'expensive' computations in ML?

$\exp$, $\sin$, $\tan$ and their inverse and otherwise related functions are transcendental, defined by an infinite power series. Meaning it takes some effort to evaluate uniformly good approximations....
Lutz Lehmann's user avatar
  • 6,109
10 votes

Why is $\exp(\ln(x))-x\neq0$ in floating point arithmetic?

The numeric precision is not perfect. You get rounding errors during your computation. When working with floats, don't check if they are = 0, but check if their absolute distance to 0 is smaller than ...
user1841833's user avatar
10 votes
Accepted

Accuracy loss in single-precision Euclidean norm computation

The squares are harmless (as long as they don't overflow/underflow), because the relative perturbation they introduce is of the order of the machine precision $u\approx 10^{-8}$. Your troubles here ...
Federico Poloni's user avatar
10 votes

Accurate computation of logbinomial(x,y)

The proposed computation of $\ln{x \choose y}$ via the logarithm of the beta function does not give rise to catastrophic cancellation other than for cases where $y=0$, or $y =x$, which are easily ...
njuffa's user avatar
  • 1,895
9 votes
Accepted

Small, unpredictable results in runs of a deterministic model

There are aspects of modern computing systems that are inherently non-deterministic that can cause these kinds of differences. As long as the differences are very small in comparison with the ...
Brian Borchers's user avatar
9 votes
Accepted

Order of operations, numerical algorithms

Let's denote by $\otimes,\oplus,\ominus$ (I was lazy trying to get circled version of division operator) the floating-point analogs of exact multiplication ($\times$), addition ($+$), and subtraction (...
Anton Menshov's user avatar
  • 8,672
9 votes
Accepted

Is it possible to proof a-b+b = a for all double floating-point numbers?

You can sometimes prove such results (or get counterexamples) using an SMT solver such as Z3 that supports floating point arithmetic. Here is a proof of a version of your theorem that says $|((x+y)-y)-...
Kirill's user avatar
  • 11.4k
8 votes
Accepted

Stabilizing a 3x3 real symmetric matrix eigenvalue calculation

This is trying to compute the eigenvalues by computing the roots of the characteristic polynomial. In this case, the characteristic polynomial is $p(t) = t^3-2t^2x$, $x=1.25\times 10^6$, and zero is a ...
Kirill's user avatar
  • 11.4k
8 votes
Accepted

Can floating point error (in FFTW3) cause non-deterministic behavior?

Non-reproducible behaviors in computing amidst different runs can involve several mechanisms, sometimes mixed. They can be especially sensitive when one iterates calculations on large sets of data, ...
Laurent Duval's user avatar
8 votes

Are BLAS implementations guaranteed to give the exact same result?

The Short Answer If the two BLAS implementations are written to carry out the operations in the exact same order, and the libraries were compiled using the same compiler flags and with the same ...
Tyler Olsen's user avatar
  • 1,512
8 votes
Accepted

Integer operations vs floating point operations

There is a nice discussion on StackOverflow regarding floating point vs integer operations. In short, the performance of the operations depends a lot on processor architecture how the data is stored ...
Anton Menshov's user avatar
  • 8,672
8 votes

Is half precision supported by modern architecture?

In my opinion, not very uniformly. Low precision arithmetic seems to have gained some traction in machine learning, but there's varying definitions for what people mean by low precision. There's the ...
rchilton1980's user avatar
  • 4,936
8 votes

Is half precision supported by modern architecture?

The accepted answer provides an overview. I'll add a few more details about support in NVIDIA processors. The support I'm describing here is 16 bit, IEEE 754 compliant, floating point arithmetic ...
Robert Crovella's user avatar
8 votes

What are some good strategies to test a floating point arithmetic implementation for double numbers?

An online search shows various floating point test suites supporting double precision (64-bit IEEE 754) that are more comprehensive than randomized testing. I have not tested any of these myself. ...
qwr's user avatar
  • 181
8 votes
Accepted

Polynomial approximation for floating-point arithmetic

The sine is an odd function, so you want that also in an approximation. A polynomial with $p(0)=0$ can be factored as $p(x)=xq(x)$, so $q(x)\approx \frac{\sin(x)}{x}$. Each interval $[2^n,2^{n+1})$ ...
Lutz Lehmann's user avatar
  • 6,109
8 votes
Accepted

How to design a sin and an arcsin function such that arcsin(sin(x))=x, where x is a finite precision floating point number

That seems impossible to do with a small numerical error, because of cardinality reasons. Consider for instance the much simpler case of the function $f(x) = x^2$ over $[0,1]$. This function maps $[0,...
Federico Poloni's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible