We’re rewarding the question askers & reputations are being recalculated! Read more.

# Questions tagged [precision]

Issues related to the representation of numerical quantities in a finite representation in a given base differing from their exact mathematical value.

64 questions
Filter by
Sorted by
Tagged with
566 views

### What is Precision-Recall Curve? [closed]

I have a data mining assignment where I make a content-based image retrieval system. I have 20 images of 5 animals. So in total 100 images. My code returns the 10 most relevant images to an input ...
3k views

### Number of equations and precision of SciPy's integrate.odeint()

Is there any reason why SciPy's integrate.odeint() should become less precise when the number of equations increase? I'm trying to solve these two sets of differential equations: \$\frac{dy_1}{dx} = ...
118 views

### Odd accuracy barrier in C/PETSc regarding finite elements

I’m implementing a finite element code (translating from a working MATLAB version, so I have results to compare to) and for some odd reason, some of my computations are only accurate to around 6 ...
837 views

### Scientific computing with Python with modern GPUs with double precision

Has anyone here used double precision scientific computing with new generation (e.g. K20) GPUs through Python? I know that this technology is rapidly evolving, but what is the best way to do this ...
1k views

### Determine the step size in a differential equation numerical solver

How can we define the precision we require in a numerical differential equation solver? What is it that I have to optimize to know? And how do I know that I'm at a sufficient time-step value? For ...
5k views

### Machine epsilon (eps)

The wiki for machine epsilon says: "Machine epsilon gives an upper bound on the relative error due to rounding in floating point arithmetic" If machine epsilon is the upper bound on the relative ...
2k views

### Matlab output in scientific notation [ERROR: Subscript indices must either be real positive integers or logicals.]

I have X, Y and Z variables in matrix form, each of size n x 1. Eg.: X = [-38.0400, -38.6700, -38.9300, -39.4500...] Whenever I run the code below: ...
3k views

### What's the state-of-the-art in highly oscillatory integral computation?

What's the state-of-the-art in the approximation of highly oscillatory integrals in both one dimension and higher dimensions to arbitrary precision?
2k views

### Need for quad precision in scientific computing?

Even if quad precision is not directly supported by most CPUs, many Compilers (GNU, Intel) support them. Also some software packages allow to compile with quad precision, e.g. PETSc. But is there ...
716 views

### Higher precision floating-point arithmetic in numerical PDE

I have the impression, from very different resources and talks with researches, that there is a growing demand for high precision computations in numerical partial differential equations. Here, high ...
454 views

### Does there exist an arbitrary-precision convex optimization solver?

I have a relatively simple convex optimization problem that involves less than 100 variables but contains a terribly ill-conditioned matrix. I have tried CVX and CPLEX; even though both can typically ...
721 views

### Diagonalization of Dense Ill Conditioned Matrices

I am trying to diagonalize some dense, ill-conditioned matrices. In machine precision, results are inaccurate (returning negative eigenvalues, eigenvectors do not have the expected symmetries). I ...