Single versus double floating-point precision

Question

Single precision floating point numbers take up half the memory and on modern machines (even on GPUs it seems) operations can be done with them at almost twice the speed compared to double precision. Many FDTD codes that I have found exclusively use single precision arithmetic and storage. Is there a rule of thumb of when it is acceptable to use single precision for solving large-scale sparse systems of equations? I assume it must heavily depend on the matrix condition number.

Furthermore, is there any effective technique which uses double precision where necessary and single where the accuracy of double is not required. For instance, I would think that for a matrix vector multiplication or a vector dot product, it might be a good idea to accumulate the results in a double precision variable (to avoid cancellation error), but that individual entries to be multiplied with each other can be multiplied using single precision.

Do modern FPU's seamlessly allow conversion from single precision (float) to double precision (double) and vice versa? Or are these costly operations?

Answer 1

For all non-trivial problems (i.e., for those where performance matters) almost all of the memory you have will be in the matrix, and relatively little in vectors. For example, for 3d Taylor-Hood elements for the Stokes equation, you have a few hundred elements per row in the matrix, and this vastly outweighs the amount of memory needed for vectors. We have thus played with the idea of storing the matrix as floats and the vectors as doubles. I don't recall our timing results but I know for sure that we haven't seen any problems with round-off etc. So this approach definitely works.

Answer 2

A nice paper on this topic is Accelerating scientific computations with mixed precision algorithms.

Answer 3

My advice would be to focus mainly on the memory consumption for the decision when to use single precision (float). So the bulk data for a FDTD computation should use float, but I would keep the problem description itself (like geometry, material parameters, excitation conditions) and all related meta data in double.

I would keep all performance uncritical and not deeply analyzed computations in double. Especially, I would keep polygonal data and other description of geometry in double (perhaps even integer if possible), since experience tells that you will never get the computational geometric parts of your code fully robust, even if it would be possible in theory.

A third part I would keep in double are analytic computations, including shortcuts using non-symmetric eigenvalue decompositions. As an example, I have a piecewise defined rotational symmetric 2D functions, and I need its Fourier transform. There would be various numerical ways involving FFTs and appropriate "analytical low pass filters" to get it "efficiently". Because it's performance uncritical, I used an "exact" analytical expression involving Bessel functions instead. Since this was a shortcut to begin with, and I won't spend any time analyzing the error propagation of my complicated formula, I better use double precision for that computation. (It still turned out that only some of the analytical equivalent expressions for the formula were usable, because some lost accuracy too quick even with double precision).