5
$\begingroup$

I am working on a problem where I need to solve approximately 500 Million Linear Regressions (OLS). What would be the most efficient way to do this (e.g. using GPU or a some framework that can do this efficiently)?

Note that in my case each individual OLS problem is "small" in size e.g. each input dataset has 100 observations with (at most) 5 features.

Here's what I have tried:

I use a 25 core machine and parallelize my computation across all cores using Python3 multiprocessing module. To solve OLS, I use numpy.linalg.solve as I found that to be more efficient than sklearn.linear_model.LinearRegression.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Are any of the input datasets overlapping? Or is none of the 100 observations for some OLS problem never seen again? I don’t have any practical experience with the multiprocessing module but if that’s efficient enough, then your solution seems reasonable for a first try. I would be curious how it compares to a C++ implementation using OpenMP or something though. $\endgroup$
    – spektr
    Commented Jul 29, 2021 at 4:14

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.