Increasing computational performance by using 16 bit numbers

Question

I recently found the following article where it was stated that using 16 bit numbers can be used to increase the computational performance of AI applications. According to the article numbers above 16 bit must be scaled to fit into the 16 bits.
Could this technique also be applied to solutions of linear system using iterative methods? How could a scaling from 64 bit numbers to 16 bit numbers look like if the biggest number which can occur in the method is not known in advance?

Answer 1

There has been considerable recent interest in numerical linear algebra using mixed precision with some combination of 16, 32, 64, and 128 bit floating point arithmetic.

For example, a low precision factorization of a matrix can be used used to precondition a higher precision iterative solution. Since the factorization takes $O(N^{3})$ operations and the iterative method takes $O(N^2)$ operations per iteration, it might be effective to compute the factorization of the matrix in faster lower precision. It turns out that using three levels of precision (e.g. single, double, and quad precision), you can get full accuracy at the middle level of precision (e.g. double) using a lot of operations and the lowest level of precision and a very few operations at the highest precision.

Whether this is fast in practice depends a lot on the hardware you're using. On most contemporary processors from Intel and AMD, the vector processing units can do twice as many single-precision (32 bit) floating point operations per cycle as double precision (64 bit) operations.

Speed of floating point operations is not the only advantage that comes with using a less precise format. For operations that are memory bound (most level 1 and level 2 BLAS operations but not level 3 operations such as matrix-matrix multiplication and matrix factorizations), using a lower precision (e.g. single instead of double) cuts the memory bandwidth required to bring the data in from memory in half. Furthermore, if data are being stored in a cache memory, using the lower precision type effectively doubles the number of floating point numbers that can be stored in the cache.

However, hardware support for 128 bit floating point operations isn't commonly available. In practice, using 128 bit floating point arithmetic in software (e.g. using two doubles to store a quadruple precision number) is typically more than 100 times slower than 64 bit double precision arithmetic in hardware. This can make the performance of the mixed-precision scheme unacceptable.

Only a few of the most recent processor models have support for 16 bit floats. This is very useful for neural network computations and could be of some use for numerical linear algebra using mixed 16-bit, 32-bit, and 64- bit arithmetic to get single precision solutions. To get double precision solutions to a linear system you'd use a mixture of single, double, and quad precision arithmetic and 16 bit floats wouldn't be helpful.

The situation with GPU's is similar (single and double precision are common, 16 bit floats are a recent addition and quadruple precision isn't supported in hardware.) The situation is complicated by the throttling of double precision floating point on many NVIDIA GPU's.

Nick Higham and his students have a number of recent papers related to this:

http://www.ma.man.ac.uk/~higham/papers/bibbase.php