Preconditioning vs. regularization

Question

I used to be more of a numerical linear algebra and computational science person, but recently, I've crossed into stats and machine learning.

For this discussion, let's focus on matrices that are not theoretically singular, but "numerically singular," which is what I typically use to describe high condition number systems.

In numerical methods, when you have an ill-conditioned linear system $Ax=b$, you can apply a preconditioner to the linear system that (hopefully) makes the problem well-posed, resulting in solutions that (1) converge faster (2) converge more accurately.

The statistics analog for ill-conditioning appears to be "multicollinearity" (e.g., in linear regression where you also solve a linear system of the form $Ax=b$). They both, more or less, mean that the linear matrix $A$ is approximately not of full rank or approximately singular, which means the solution you obtain may not be unique or exhibits a high degree of variance for slight perturbations in the input. In statistics, the go-to method appears to be $\ell_2$ or $\ell_1$ regularization, which they refer to as "ridge" and "lasso" regression.

I am wondering why preconditioning isn't a go to method in the statistics community, and why regularization isn't a go to method in the numerical methods / scientific computing community?

My suspicion for the latter is that regularization gives you a unique solution, but you're solving an entirely different system that may not be representative of what you were originally attempting to solve, so this seems it may not be acceptable in some situation where you're modeling a physical system with an underlying mathematical model that describes the underlying physics/chemistry/etc...

Answer 1

I think there is a bit of jargon confusion here. So I will emphasize some words and the context I am using them.

In NLA, when I attempt to solve a problem (they usually come from discretizations of PDEs), I know that the problem is well-posed, i.e. has a unique solution, beforehand. The problem may be ill-conditioned (opposed to well-conditioned) and that can be alleviated by preconditioning the matrix. But if the matrix is singular, preconditioning it will not help much, if at all. There are exceptions; for example, hydrostatic pressure (the corresponding matrix is 1 fewer rank than full) where you can guarantee the uniqueness of the solution by forcing the mean to be zero. But even in that case, preconditioning is not used to fix the singularity of the problem, but to accelerate the solution.

On the other hand, regularization is used to make the problem well-posed (as opposed to ill-posed) and force the solution to be unique. However, as you mentioned, in such a case the solution to the regularized problem may not be a solution to the original problem. Which is not a big deal if you do not expect a unique solution to begin with.

Also I should add, Lagrange multipliers approach (which is commonly used in optimization and statistics) can be considered to be a form of preconditioning. But it is never considered as regularization (even though it has a similar form). So, IMO, it is false to say that preconditioning is not a go-to method in the statistics community. It is just not very useful outside a small class of problems.