Timeline for Are there tasks in machine learning which require double precision floating points?
Current License: CC BY-SA 3.0
6 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Feb 21 at 10:47 | comment | added | Federico Poloni | @XanderDunn OP refers to linear algebra problems such as least squares problems, which appear in some ML models (e.g., reservoir computing, extreme learning). I am no expert but I don't think BERT is trained using least squares, just standard stochastic gradient descent with backpropagation, am I right? Anyhow, it looks like your comment would make a good answer, if you wish to write one. | |
| Nov 29, 2020 at 1:06 | comment | added | Xander Dunn | We don't see this in practice in machine learning. Consider RoBERTa (arxiv.org/pdf/1907.11692.pdf), one of the largest and most widely used natural language models, which was trained using mixed precision, dropping all the way down to 16 bits. Consider even more radical reductions in precision here: engineering.fb.com/2018/11/08/ai-research/floating-point-math. BERT uses 32 bit precision (github.com/google-research/bert/search?q=float32) and (blog.inten.to/speeding-up-bert-5528e18bb4ea). GPT-2 uses tensorflow's default float32 (github.com/openai/gpt-2) | |
| Nov 27, 2015 at 10:52 | comment | added | David Ketcheson | The circumstances I have described above are those where you will get 100% error. If you need more than a single digit of accuracy, the situation is even worse. I'm afraid your intuition isn't very helpful here if you haven't studied numerical linear algebra. | |
| Nov 27, 2015 at 9:00 | comment | added | Marat Zakirov | I see... But I do not assume that ML is something which is needed precise solution. I'd like to see example there using floats totally mess the result. Have you ever seen cases where Neural network was used for astro-navigation, for example? Or lets look at it under another angle. All MLs are basically optimization problems. Could you imagine optimizations tasks which give totally different result (much more optimal) when used doubles? How often (and in what conditions) such cases could appear? | |
| Nov 26, 2015 at 19:13 | history | edited | David Ketcheson | CC BY-SA 3.0 |
added 69 characters in body
|
| Nov 26, 2015 at 18:58 | history | answered | David Ketcheson | CC BY-SA 3.0 |