I am not an expert in machine learning, but I can outline the considerations that are relevant.

The numerical calculations in machine learning are generally linear algebra -- either solving linear systems or linear least squares.  For both types of problems, there are well-known backward-stable methods, so I will assume you are using a backward-stable algorithm.  Then you should expect an error of roughly
$\kappa \epsilon$, where $\kappa$ is the condition number of the problem and $\epsilon$ is unit roundoff.

For the linear system $Ax=b$, you have $\kappa = \|A\| \|A^{-1}\| = \kappa(A)$, the condition number of the matrix $A$.  For the least squares problem, the condition number can fall anywhere in the range $[\kappa(A),\kappa^2(A)]$; see e.g. the text of Trefethen & Bau for details.

Thus for linear systems, single precision will be sufficient as long as $\kappa(A)$ is much less than $10^7$.  For least squares, you may already be in trouble when $\kappa(A)\approx 10^3 - 10^4$.  For large datasets, those are not very large condition numbers.