Historically, iterative methods for solving relatively simple-structured systems $Ax=b$ with $A$ being a $4\times 4$ matrix or to find the eigenvalues of that matrix assuming in both problems that $A$ is non-singular were not quite popular and were not needed in pre-1950s as stationary methods or perhaps classical iterative algorithms such as Jacobi, Gauss-Seidel would handle the task easily.
Then in the $1950$s the rapid development of computational tools made it possible to deal with large and sparse matrix system (possibly a $100\times 100$ at that time in engineering applications). This is all thanks to the development of Krylov Subspace tools which at that time was considered to be "high-tech". This is seen from various celebrated algorithms such as Arnoldi Iteration (Arnoldi-Lanczos 1950) and Conjugate Gradient (Stiefel,1952). During the 1970s and 1980s when we had tons and tons of transistors inside computers this made it possible for us to witness the birth of GMRES and BiCGSTAB which are known to handle relatively large and sparse systems.
In fact, it was during that period that BLAS and LAPACK were developed and ever since have continued to upgrade various Krylov algorithms and achieve a high level of robustness.
In machine learning, many problems such as regression, neural networks, and even recommender systems can one way or another be formulated as an optimization problem which in turn can be expressed as a linear system $Fx=z$. The apparent default choice for solving such problems is the "Gradient Descent Algorithm" (and its derivatives) a popular optimizer algorithm that seems to be used literally everywhere in machine learning problems (movie recommender system, linear regression,...) even a famous variant is the ADAM Optimizer which in many deep learning libraries is set as the default optimizer.
Given that numerical linear algebra libraries have been for the past 40 years enhancing Krylov algorithms, why haven't they made an impact in today's world of machine learning as they did in engineering communities?
Let's say linear regression is the underlying machine learning problem, why is the gradient descent algorithm preferred over using BiCGSTAB with some restart when rewriting the regression problem as $Fx=z$ where $F$ could be sparse and $10000\times 10000$?
I believe one answer (though it may be biased) is that the majority of the machine learning community is not paying attention to the computational mechanism of the algorithm. For instance, you can go ahead and apply ADAM optimizer to solve a simple regression problem (overkill) or a complicated large regression problem and may end up witnessing your algorithm diverging without properly understanding why. I found a meme from medium.com that jokingly describes this:
Update 1: I would like to thank
davidhigh for the detailed answer. I would like to add that my experience with Krylov methods were mainly focused on modelling dynamical system. Tim Davis has a very interesting collection of sparse matrices with detailed information on where the problem arises from (in particular engineering applications) Krylov methods are highly applicable there.
Neil Lindquist made an important correction regarding BLAS and LAPACK not having Krylov solvers.
Final Remark: I welcome any additional answer especially if some users have experience with using Krylov methods in ML applications.