I am working on a problem where I need to solve approximately 500 Million Linear Regressions (OLS). What would be the most efficient way to do this (e.g. using GPU or a some framework that can do this efficiently)?

Note that in my case each individual OLS problem is "small" in size e.g. each input dataset has 100 observations with (at most) 5 features.

Here's what I have tried:

I use a 25 core machine and parallelize my computation across all cores using Python3 multiprocessing module. To solve OLS, I use numpy.linalg.solve as I found that to be more efficient than sklearn.linear_model.LinearRegression.

  • 1
    $\begingroup$ Are any of the input datasets overlapping? Or is none of the 100 observations for some OLS problem never seen again? I don’t have any practical experience with the multiprocessing module but if that’s efficient enough, then your solution seems reasonable for a first try. I would be curious how it compares to a C++ implementation using OpenMP or something though. $\endgroup$
    – spektr
    Jul 29, 2021 at 4:14


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.