I used to be more of a numerical linear algebra and computational science person, but recently, I've crossed into stats and machine learning.
For this discussion, let's focus on matrices that are not theoretically singular, but "numerically singular," which is what I typically use to describe high condition number systems.
In numerical methods, when you have an ill-conditioned linear system $Ax=b$, you can apply a preconditioner to the linear system that (hopefully) makes the problem well-posed, resulting in solutions that (1) converge faster (2) converge more accurately.
The statistics analog for ill-conditioning appears to be "multicollinearity" (e.g., in linear regression where you also solve a linear system of the form $Ax=b$). They both, more or less, mean that the linear matrix $A$ is approximately not of full rank or approximately singular, which means the solution you obtain may not be unique or exhibits a high degree of variance for slight perturbations in the input. In statistics, the go-to method appears to be $\ell_2$ or $\ell_1$ regularization, which they refer to as "ridge" and "lasso" regression.
I am wondering why preconditioning isn't a go to method in the statistics community, and why regularization isn't a go to method in the numerical methods / scientific computing community?
My suspicion for the latter is that regularization gives you a unique solution, but you're solving an entirely different system that may not be representative of what you were originally attempting to solve, so this seems it may not be acceptable in some situation where you're modeling a physical system with an underlying mathematical model that describes the underlying physics/chemistry/etc...