# Preconditioning vs. regularization

I used to be more of a numerical linear algebra and computational science person, but recently, I've crossed into stats and machine learning.

For this discussion, let's focus on matrices that are not theoretically singular, but "numerically singular," which is what I typically use to describe high condition number systems.

In numerical methods, when you have an ill-conditioned linear system $$Ax=b$$, you can apply a preconditioner to the linear system that (hopefully) makes the problem well-posed, resulting in solutions that (1) converge faster (2) converge more accurately.

The statistics analog for ill-conditioning appears to be "multicollinearity" (e.g., in linear regression where you also solve a linear system of the form $$Ax=b$$). They both, more or less, mean that the linear matrix $$A$$ is approximately not of full rank or approximately singular, which means the solution you obtain may not be unique or exhibits a high degree of variance for slight perturbations in the input. In statistics, the go-to method appears to be $$\ell_2$$ or $$\ell_1$$ regularization, which they refer to as "ridge" and "lasso" regression.

I am wondering why preconditioning isn't a go to method in the statistics community, and why regularization isn't a go to method in the numerical methods / scientific computing community?

My suspicion for the latter is that regularization gives you a unique solution, but you're solving an entirely different system that may not be representative of what you were originally attempting to solve, so this seems it may not be acceptable in some situation where you're modeling a physical system with an underlying mathematical model that describes the underlying physics/chemistry/etc...

• I guess this question, at least at a basic level, targets the difference between the pseudoinverse and L2-regularization (aka ridge regression). The details are given in Hastie, but basically they ate doing the same things -- ruling out collinear dimensions -- in a slightly different way. – davidhigh Mar 21 at 18:01

I think there is a bit of jargon confusion here. So I will emphasize some words and the context I am using them.

In NLA, when I attempt to solve a problem (they usually come from discretizations of PDEs), I know that the problem is well-posed, i.e. has a unique solution, beforehand. The problem may be ill-conditioned (opposed to well-conditioned) and that can be alleviated by preconditioning the matrix. But if the matrix is singular, preconditioning it will not help much, if at all. There are exceptions; for example, hydrostatic pressure (the corresponding matrix is 1 fewer rank than full) where you can guarantee the uniqueness of the solution by forcing the mean to be zero. But even in that case, preconditioning is not used to fix the singularity of the problem, but to accelerate the solution.

On the other hand, regularization is used to make the problem well-posed (as opposed to ill-posed) and force the solution to be unique. However, as you mentioned, in such a case the solution to the regularized problem may not be a solution to the original problem. Which is not a big deal if you do not expect a unique solution to begin with.

Also I should add, Lagrange multipliers approach (which is commonly used in optimization and statistics) can be considered to be a form of preconditioning. But it is never considered as regularization (even though it has a similar form). So, IMO, it is false to say that preconditioning is not a go-to method in the statistics community. It is just not very useful outside a small class of problems.

• Wait, sorry, I'm confused. What is the jargon confusion? I think everything you said are things I said and agree with. – anonuser01 Mar 18 at 20:41
• My apologies, maybe I misunderstood. You said "you can apply a preconditioner to the linear system that (hopefully) makes the problem well-posed" and well-posedness is not the reason why you would use a preconditioner. It may be a typo on your part or it may be me being pedantic. Either case, I apologize. – Abdullah Ali Sivas Mar 18 at 20:51
• Oh yeah I see what you're saying. So I should clarify (and I'll edit that into the OP) that I'm only referring to non-singular matrices here. The matrix can be "numerically" or "approximately" singular even though it's not actually theoretically singular. – anonuser01 Mar 18 at 20:52
• Note that it some situations it's possible to bound the bias introduced by regularization. – Brian Borchers Mar 19 at 4:28
• Let $A_1x_1=b$ be the regularized version of the problem $Ax=b$. Under some conditions, it is possible to bound $\|x_1-x\|$ in terms of the regularization parameter. @BrianBorchers would probably know more about this, I have seen it in a talk and it was honestly interesting. – Abdullah Ali Sivas Mar 19 at 16:10