9
$\begingroup$

Double-precision calculations are significantly slower or more expensive than single-precision calculations. For example, the NVidia Tesla which performs well on doubles is much more expensive then regular GPU.

At the same time I do not know about machine learning cases where double-precision is really needed. Currently I think that 64-bit floats are needed by the model only in case then it is over-fitted. In other words doubles are needed only by "bad" models with bad generalization.

$\endgroup$
6
  • 4
    $\begingroup$ What kind of machine learning? SVD/PCA-based? Surely the thing to do would be to test on a sample dataset and compare the difference between single and double precision, and then decide whether that is an appropriate error? $\endgroup$ Commented Nov 26, 2015 at 11:10
  • $\begingroup$ Testing is always a good thing but I do not understand in theory why any body could even need doubles. $\endgroup$ Commented Nov 26, 2015 at 12:54
  • $\begingroup$ This is just blowing my brain in CrossValidated community I was said that people from computational science will be more appropriate people to ask. $\endgroup$ Commented Nov 26, 2015 at 13:37
  • 1
    $\begingroup$ An excellent exercise to understand this is adding up the numbers between 0 and 10^8 first in sorted then random order. You will discover if you do this in single a precision float you will get different answers. Next realize that any time a vector spans these orders of magnitude, you will now be dependent on its ordering to get a correct sum. This means that your ML algo may function differently given a different ordering of the same data ... probably an undesirable trait. $\endgroup$
    – meawoppl
    Commented Nov 29, 2015 at 4:11
  • 3
    $\begingroup$ Take a look at the literature on PINNs. These models need to be trained to reasonably high accuracy in order to get results that actually satisfy the underlying PDE. $\endgroup$ Commented Feb 8 at 23:50

3 Answers 3

12
$\begingroup$

I am not an expert in machine learning, but I can outline the considerations that are relevant.

The numerical calculations in machine learning are generally linear algebra -- either solving linear systems or linear least squares. For both types of problems, there are well-known backward-stable methods, so I will assume you are using a backward-stable algorithm. Then you should expect an error of roughly $\kappa \epsilon$, where $\kappa$ is the condition number of the problem and $\epsilon$ is unit roundoff.

For the linear system $Ax=b$, you have $\kappa = \|A\| \|A^{-1}\| = \kappa(A)$, the condition number of the matrix $A$. For the least squares problem, the condition number can fall anywhere in the range $[\kappa(A),\kappa^2(A)]$; see e.g. the text of Trefethen & Bau for details.

Thus for linear systems, single precision will be sufficient as long as $\kappa(A)$ is much less than $10^7$. For least squares, you may already be in trouble when $\kappa(A)\approx 10^3 - 10^4$. For large datasets, those are not very large condition numbers. So it certainly seems plausible that you may need double precision.

$\endgroup$
4
  • $\begingroup$ I see... But I do not assume that ML is something which is needed precise solution. I'd like to see example there using floats totally mess the result. Have you ever seen cases where Neural network was used for astro-navigation, for example? Or lets look at it under another angle. All MLs are basically optimization problems. Could you imagine optimizations tasks which give totally different result (much more optimal) when used doubles? How often (and in what conditions) such cases could appear? $\endgroup$ Commented Nov 27, 2015 at 9:00
  • 3
    $\begingroup$ The circumstances I have described above are those where you will get 100% error. If you need more than a single digit of accuracy, the situation is even worse. I'm afraid your intuition isn't very helpful here if you haven't studied numerical linear algebra. $\endgroup$ Commented Nov 27, 2015 at 10:52
  • $\begingroup$ We don't see this in practice in machine learning. Consider RoBERTa (arxiv.org/pdf/1907.11692.pdf), one of the largest and most widely used natural language models, which was trained using mixed precision, dropping all the way down to 16 bits. Consider even more radical reductions in precision here: engineering.fb.com/2018/11/08/ai-research/floating-point-math. BERT uses 32 bit precision (github.com/google-research/bert/search?q=float32) and (blog.inten.to/speeding-up-bert-5528e18bb4ea). GPT-2 uses tensorflow's default float32 (github.com/openai/gpt-2) $\endgroup$ Commented Nov 29, 2020 at 1:06
  • $\begingroup$ @XanderDunn OP refers to linear algebra problems such as least squares problems, which appear in some ML models (e.g., reservoir computing, extreme learning). I am no expert but I don't think BERT is trained using least squares, just standard stochastic gradient descent with backpropagation, am I right? Anyhow, it looks like your comment would make a good answer, if you wish to write one. $\endgroup$ Commented Feb 21 at 10:47
2
$\begingroup$

Perhaps it's better to say that it depends on the nature (details) of the algorithm and linear algebra implementation.

More critically, while deep neural networks (DNN) have become widely popular, there are many other valuable techniques offered by machine learning. Thus, this question requires an answer that caters to both - ML in general and DNN in specifics.

Additionally, the paradigms of machine learning can be divided into practitioners and scholars. Practitioners are concerned with delivering task performance, whereas scholars are interested in seeking answers and giving advices via investigations and numerical analysis. Scholars may apply numerical techniques to machine learning algorithms in ways that practitioners won't; this is why double-precision may sometimes be involved in scholastic research.

Deep neural network, in practice, does not involve matrix inversion at all. The main algorithm involved is backpropagation. The gradient is computed after each forward pass; it is then scaled with the step size parameter and added back to the parameter matrices on each layer. Typically single precision is good for training, and inference can do with very low precisions. Typically the "adaptive mixed precision" technique is used in DNN. Typically, double precision does not improve DNN task accuracy at all when compared to single precision.

That said, singular value decomposition (SVD) is an option (choice) in the implementation of a technique called Low-Rank Adaptation (LoRA), the latter being well known for cost-effective fine-tuning of deep neural network large language models (closely related to the Transformer architecture). Rather than performing the steps in a high floating point precision, much of the engineering innovation went into modifying the SVD algorithm such that a reasonable approximation can be derived using the lowest possible floating point precision. In other words, whenever a speed-accuracy tradeoff happens, a lot of engineering effort is committed in order to enable a variety of middle ground approaches.

Outside of DNN, there are still many other machine learning algorithms that are not based on neural networks, and those might need to use a higher precision because they are based on different linear algebra techniques. The least squares method is probably the best well-known example.

Another relevant concern is numerical reproducibility. In DNN, numerical reproducibility is not highly valued. Instead, practitioners are fundamentally more concerned about replicating the task performance (accuracy) when a model is trained with a specific data set, a carefully described DNN architecture, and a detailed description of the training process. Training with stochastic gradient descent (SGD), minibatch, and data augmentation (data perturbations and alterations intended to make the model more task-robust) already involves tons of randomization; DNN practitioners are far more concerned about these stochastic techniques leading to a DNN model with good task performance (e.g. correctly classifying an object and its bounding box, without requiring subpixel accuracy) than the numerical reproducibility of the trained model parameters.

I've heard that KL divergence can be used to find an accurate approximation between a layer implemented with higher precision and a layer with lower precision, even when nonlinearity (e.g. multiple layers of DNN) is involved. This technique is often implemented into DNN frameworks to allow speed-accuracy tradeoffs. Unfortunately, I'm not familiar with these details.

$\endgroup$
0
$\begingroup$

I think you're right that double precision is not needed usually.

As a proxy, Apple's Metal framework doesn't support doubles at all, and it's being billed as a compute framework with Apple-provided functions for machine learning.

Info about Metal data types: https://developer.apple.com/metal/Metal-Shading-Language-Specification.pdf

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.