I am working on a problem where I need to solve approximately 500 Million Linear Regressions (OLS). What would be the most efficient way to do this (e.g. using GPU or a some framework that can do this efficiently)?
Note that in my case each individual OLS problem is "small" in size e.g. each input dataset has 100 observations with (at most) 5 features.
Here's what I have tried:
I use a 25 core machine and parallelize my computation across all cores using Python3 multiprocessing module. To solve OLS, I use numpy.linalg.solve as I found that to be more efficient than sklearn.linear_model.LinearRegression.