Suppose $X$ is a dense $m\times n$ data matrix and we seek to find $w$ by approximately iterating least-squares filter equation:
$$w = w - \mu (X'X)^{-1}X'(Xw-b)$$
What are known approaches for $10^9<m<\infty$ and $n\approx 10^6$?
This setting comes up in Machine Learning and people tend to invent their own techniques for approximate pseudo-inverse of $X$, which in practice has low stable-rank, without referencing scientific computing literature.
For instance KFAC method addresses large $n$ by writing $x\approx a\otimes b$ and using this factorization to approximate $(X'X)^{-1}\approx (A'A)^{-1}\otimes (B'B)^{-1}$. Making this update numerically stable requires considerable tweaking.
Any literature pointers appreciated!
Edit Jul 19 crossposted on mathoverflow