5
$\begingroup$

Double-precision calculations are significantly slower or more expensive than single-precision calculations. For example, the NVidia Tesla which performs well on doubles is much more expensive then regular GPU.

At the same time I do not know about machine learning cases where double-precision is really needed. Currently I think that 64-bit floats are needed by the model only in case then it is over-fitted. In other words doubles are needed only by "bad" models with bad generalization.

$\endgroup$
  • 1
    $\begingroup$ What kind of machine learning? SVD/PCA-based? Surely the thing to do would be to test on a sample dataset and compare the difference between single and double precision, and then decide whether that is an appropriate error? $\endgroup$ – dr.blochwave Nov 26 '15 at 11:10
  • $\begingroup$ Testing is always a good thing but I do not understand in theory why any body could even need doubles. $\endgroup$ – Marat Zakirov Nov 26 '15 at 12:54
  • $\begingroup$ This is just blowing my brain in CrossValidated community I was said that people from computational science will be more appropriate people to ask. $\endgroup$ – Marat Zakirov Nov 26 '15 at 13:37
  • $\begingroup$ An excellent exercise to understand this is adding up the numbers between 0 and 10^8 first in sorted then random order. You will discover if you do this in single a precision float you will get different answers. Next realize that any time a vector spans these orders of magnitude, you will now be dependent on its ordering to get a correct sum. This means that your ML algo may function differently given a different ordering of the same data ... probably an undesirable trait. $\endgroup$ – meawoppl Nov 29 '15 at 4:11
  • $\begingroup$ I don't understand the remark about the Nvidia Tesla. Picking a particular model of a particular brand of GPU to demonstrate that double precision arithmetic is expensive computationally because it costs more dollars seems odd. $\endgroup$ – Spaceghost Nov 30 '15 at 18:59
11
$\begingroup$

I am not an expert in machine learning, but I can outline the considerations that are relevant.

The numerical calculations in machine learning are generally linear algebra -- either solving linear systems or linear least squares. For both types of problems, there are well-known backward-stable methods, so I will assume you are using a backward-stable algorithm. Then you should expect an error of roughly $\kappa \epsilon$, where $\kappa$ is the condition number of the problem and $\epsilon$ is unit roundoff.

For the linear system $Ax=b$, you have $\kappa = \|A\| \|A^{-1}\| = \kappa(A)$, the condition number of the matrix $A$. For the least squares problem, the condition number can fall anywhere in the range $[\kappa(A),\kappa^2(A)]$; see e.g. the text of Trefethen & Bau for details.

Thus for linear systems, single precision will be sufficient as long as $\kappa(A)$ is much less than $10^7$. For least squares, you may already be in trouble when $\kappa(A)\approx 10^3 - 10^4$. For large datasets, those are not very large condition numbers. So it certainly seems plausible that you may need double precision.

$\endgroup$
  • $\begingroup$ I see... But I do not assume that ML is something which is needed precise solution. I'd like to see example there using floats totally mess the result. Have you ever seen cases where Neural network was used for astro-navigation, for example? Or lets look at it under another angle. All MLs are basically optimization problems. Could you imagine optimizations tasks which give totally different result (much more optimal) when used doubles? How often (and in what conditions) such cases could appear? $\endgroup$ – Marat Zakirov Nov 27 '15 at 9:00
  • 2
    $\begingroup$ The circumstances I have described above are those where you will get 100% error. If you need more than a single digit of accuracy, the situation is even worse. I'm afraid your intuition isn't very helpful here if you haven't studied numerical linear algebra. $\endgroup$ – David Ketcheson Nov 27 '15 at 10:52
0
$\begingroup$

I think you're right that double precision is not needed usually.

As a proxy, Apple's Metal framework doesn't support doubles at all, and it's being billed as a compute framework with Apple-provided functions for machine learning.

Info about Metal data types: https://developer.apple.com/metal/Metal-Shading-Language-Specification.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.