2
$\begingroup$

Moore's law states that the number of transistors on an integrated circuit grow exponentially, roughly doubling at a period of 20 months. This affects the amount of memory available and the speed of computation, which roughly double at the same rate.

This also affects the speed of floating-point computation, if the precision is fixed. But I wonder whether one can observe a similar growth in the precision available (for scientific computing).

As a follow-up, what would such growth imply for scientific computing? For example, if we regard a linear systems of equations $Ax = b$, then the condition number of the matrix $A$ describes how strong the solution $x$ is affected if we successively drop decimal places of $b$. The larger the condition number, the more change in $x$, the worse for computing. In the reverse direction, any increase in precision would naturally lead to an improved exactness in the solution, at rate a proportional to the condition number, which were a good thing. How can this be interpreted?

PS: I have found engineering literature that, similarly, warns the reader that, e.g., a high convergence rate in numerical methods, i.e. $O(h^t)$ with large $t$, on the other hand means a strong detoriation if $h$ is increased, where $h$ denotes step-size or mesh-size in the usual contexts. This is similar to my point of view on the condition number (but may be equally wrong).

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
10
$\begingroup$

I honestly do not think there is a very strong analogy here: double precision has been the standard since 1985 (IEEE 754-1985), and there are libraries that can compute in arbitrary precision, e.g., GMP, although there is a significant drop in the number of floating point operations that can be computed per second (flops). If one was to try to draw any analogy to Moore's law, it would have to be limited to hardware-based floating point precision, and the timeframe for the doubling would at least be 25 years.

I am very much in favor of efficient high precision computation becoming more standard, but higher precision only reduces one of the sources of error; for example, small perturbations in the formation of the right-hand side could still be catastrophic.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ +1. To add another argument: if there were much demand for higher accuracy, then relatively high performance double-double (or quad-double) libraries such as David Bailey's would be very popular as a low-impact step up from regular double precision (at least, substantially lower-impact than GMP, because allocation is so much simpler). Since they are only moderately popular, there must not be too much demand. $\endgroup$
    – Erik P.
    Jan 4, 2012 at 21:46
  • 2
    $\begingroup$ @ErikP. I would actually argue that double-double and friends are not used more often simply because it greatly complicates library developers' jobs. For instance, BLAS and LAPACK do not currently support higher precision, and neither does MPI. This leads to difficult design decisions. Not surprisingly, PETSc has one of the most sophisticated build systems in existence, and PETSc supports quad precision. $\endgroup$
    – Jack Poulson
    Jan 4, 2012 at 21:50
5
$\begingroup$

As far as perspective goes for scientific computing, it's not clear to me that additional precision in floating point computation would matter beyond a certain point. The most precise calculations of physical constants from measurements tend to be accurate to 10 or 11 significant figures, and at most 14. Typically, that level of precision isn't necessary even for scientific work, unless you're working at a place NIST or CERN, or in certain subdisciplines of physics. It is possible to represent that level of precision in inputs using double precision arithmetic; after that, accuracy of calculations depends on a variety of factors. I think it's important to keep the application in mind, however; there's no point in obsessing about accuracy to 10 decimal places if you can't measure what you're modeling to more than a part in 1,000.

As @JackPoulson points out, all increasing the level of precision does is decrease unit roundoff, which is one factor in error bounding results like those found in Accuracy and Stability of Numerical Algorithms (second edition) by Nick Higham. It is not a replacement for good practices in numerical analysis like proper scaling of models, good preconditioners, good choice of algorithms (such as using a QR factorization for least squares problems instead of solving the normal equations). The link above to page 17 of Higham's book provides a more thorough discussion of what increasing the precision gives you in terms of accuracy; he shows an example where increasing the precision does not yield any additional accuracy at all.

Improve this answer
$\endgroup$
7
  • $\begingroup$ I actually do not agree with your first paragraph; there are most certainly cases where roundoff error is the source of the problem, even when the original linear system is known exactly (consider trying to solve against a Hilbert matrix). That Higham has presented a case where extra precision does not help does not imply that it never helps. $\endgroup$
    – Jack Poulson
    Jan 4, 2012 at 16:59
  • $\begingroup$ I agree with you; I don't mean to imply that extra precision never helps. Barrier methods in optimization are useless in single precision, but useful in double precision; roundoff was a problem there. The question is, what does the cost of that precision in memory and FLOPS buy you? Higher accuracy intermediate results? You can't measure most things to double precision, so any final results for physical models in quad precision are going to waste at least 10 sig figs. We can't even necessarily estimate the error in our double precision calculations right now, beyond simple cases. $\endgroup$
    – Geoff Oxberry
    Jan 4, 2012 at 17:14
  • 1
    $\begingroup$ Suppose that you knew you wanted to solve against an analytically known matrix (such as the Hilbert matrix) which had a condition number of 10^20. Even if your right-hand side is also exact, it is possible to get zero digits correct in the floating-point solution in double-precision. $\endgroup$
    – Jack Poulson
    Jan 4, 2012 at 17:18
  • $\begingroup$ I agree. Let me clarify (and edit). I'm trying to point out that there are diminishing returns. Suppose you could calculate to arbitrary precision cheaply. As long as the first 10-ish significant figures are accurate, that's all that matters for most physics. There gets to be some point where additional precision just does not matter when compared to the measurable precision of results. Having never needed more than double precision in my own work, and having never heard other people clamor for quad precision, I don't see a strong motivation for it. $\endgroup$
    – Geoff Oxberry
    Jan 4, 2012 at 18:32
  • 1
    $\begingroup$ @Martin: I mean in the sense that there is no perturbation before conversion into float-point format, e.g., from measurement or quadrature error. $\endgroup$
    – Jack Poulson
    Jan 4, 2012 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.