# Questions tagged [precision]

Issues related to the representation of numerical quantities in a finite representation in a given base differing from their exact mathematical value.

91 questions
Filter by
Sorted by
Tagged with
9k views

### Is half precision supported by modern architecture?

I am new to computer science and I was wondering whether half precision is supported by modern architecture in the same way as single or double precision is. I thought the 2008 revision of IEEE-754 ...
• 403
6k views

### How can the gravitational n-body problem be solved in parallel?

How can the gravitational n-body problem be solved numerically in parallel? Is precision-complexity tradeoff possible? How does precision influence the quality of the model?
• 383
4k views

### What's the state-of-the-art in highly oscillatory integral computation?

What's the state-of-the-art in the approximation of highly oscillatory integrals in both one dimension and higher dimensions to arbitrary precision?
• 383
5k views

### Numerically stable way of computing angles between vectors

When applying the classical formula for the angle between two vectors: $$\alpha = \arccos \frac{\mathbf{v_1} \cdot \mathbf{v_2}}{\|\mathbf{v_1}\| \|\mathbf{v_2}\|}$$ one finds that, for very small/...
3k views

### How can I avoid catastrophic cancellation?

I have the following formula that I need to rewrite in order to avoid catastrophic cancellation. $$y =\sqrt{\frac{1}{2}\left(1-\sqrt{1-x^{2}}\right)}$$ As $x$ becomes smaller, $\sqrt{1-x^{2}}$ ...
• 273
5k views

### Single versus double floating-point precision

Single precision floating point numbers take up half the memory and on modern machines (even on GPUs it seems) operations can be done with them at almost twice the speed compared to double precision. ...
• 1,320
944 views

### Scientific computing with Python with modern GPUs with double precision

Has anyone here used double precision scientific computing with new generation (e.g. K20) GPUs through Python? I know that this technology is rapidly evolving, but what is the best way to do this ...
• 516
2k views

### Why can ill-conditioned linear systems be solved precisely?

According to the answer here, large condition number (for linear system solving) decreases the guaranteed number of correct digits in the floating point solution. Higher order differentiation matrices ...
• 832
1k views

### Higher precision floating-point arithmetic in numerical PDE

I have the impression, from very different resources and talks with researches, that there is a growing demand for high precision computations in numerical partial differential equations. Here, high ...
• 3,680
2k views

### Numerical derivative and finite difference coefficients: any update of the Fornberg method?

When one want to compute numerical derivatives, the method presented by Bengt Fornberg here (and reported here) is very convenient (both precise and simple to implement). As the original paper date ...
• 343
858 views

### Diagonalization of Dense Ill Conditioned Matrices

I am trying to diagonalize some dense, ill-conditioned matrices. In machine precision, results are inaccurate (returning negative eigenvalues, eigenvectors do not have the expected symmetries). I ...
• 101
458 views

### Small, unpredictable results in runs of a deterministic model

I have a sizable model (~5000 lines) written in C. It is a serial program, with no random number generation anywhere. It makes use of the FFTW library for functions using FFT - I do not know the ...
• 193
4k views

### Are there tasks in machine learning which require double precision floating points?

Double-precision calculations are significantly slower or more expensive than single-precision calculations. For example, the NVidia Tesla which performs well on doubles is much more expensive then ...
265 views

### Representing Eisenstein numbers without floats

I have a project where I need to use quadratic fields Specifically numbers of the form $a + b \sqrt{-3}$ with $a,b \in \mathbb{Q}$. For example here are the prime numbers in Eisenstein integers: I ...
• 927
534 views

### Updating matrix diagonal with Woodbury matrix identity and maintaining numerical accuracy

I have a dense matrix A and its corresponding inverse $A^{-1}$. The Woodbury matrix identity states: $$(A + UCV)^{-1} = A^{-1} - A^{-1}U(C^{-1} + VA^{-1}U)^{-1}VA^{-1}$$ I wish to perform small ...
• 1,320
833 views

### What algorithm for solving a set of stiff ODEs would be easiest to port to high precision floating point arithmetic?

I want to solve a relatively small system of stiff ODEs (< 10 first-order equations) using high precision floating point arithmetic (using MPFR or alike). What would be the easiest algorithm to ...
• 6,159
150 views

### How can I numerically solve an ODE to $N$ provably correct digits?

Suppose we have an initial value problem of the form $$\frac{\mathrm{d} \mathbf{x}}{\mathrm{d} t} = f(\mathbf{x}) \qquad \mathbf{x}(0) = \mathbf{x}_0$$ where $\mathbf{x}_0 \in \mathbb{R}^n$ is known ...
• 223
2k views

### Need for quad precision in scientific computing?

Even if quad precision is not directly supported by most CPUs, many Compilers (GNU, Intel) support them. Also some software packages allow to compile with quad precision, e.g. PETSc. But is there ...
• 800
996 views

• 1,739
120 views

### Summation of trigonometric functions results in error with finite precision

Consider the following expression: $$f(t) = B+\sum_{k=1}^{N} A_k\cos(\omega_kt)$$ where $A$ and $B$ are known. the frequencies are also known but are not multiples of a fundamental frequency. However, ...
431 views

• 149