I am working on a problem where I need to solve approximately 500 Million Linear Regressions (OLS). What would be the most efficient way to do this (e.g. using GPU or a some framework that can do this efficiently)?

Note that in my case each individual OLS problem is "small" in size e.g. each input dataset has 100 observations with (at most) 5 features.

Here's what I have tried:

I use a 25 core machine and parallelize my computation across all cores using Python3 multiprocessing module. To solve OLS, I use numpy.linalg.solve as I found that to be more efficient than sklearn.linear_model.LinearRegression.

  • $\begingroup$ Are any of the input datasets overlapping? Or is none of the 100 observations for some OLS problem never seen again? I don’t have any practical experience with the multiprocessing module but if that’s efficient enough, then your solution seems reasonable for a first try. I would be curious how it compares to a C++ implementation using OpenMP or something though. $\endgroup$
    – spektr
    Jul 29 at 4:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.