I want to preprocess a relatively large dataset using python. I implemented some Dask parallelization and was stunned by the time reduction. I figure there are other libraries or frameworks I could use : multiprocessing, Numba, joblib, maybe even PyOpenCL (I don't have a CUDA GPU)…

Are those libraries alternatives of each other, or can one expect a significant improvement by mixing them all ? Are there common guidelines about when to use one or the other ?

I already read mixing Dask and Numba was way more efficient than using either of them.

  • $\begingroup$ FWIW, my use case machine runs an Intel i5-4200U (2 cores, 4 threads), with integrated graphics. $\endgroup$ – Skippy le Grand Gourou Feb 5 at 21:16

Dask schedules tasks across processes and across nodes, so it is appropriate for use on a single computer, supercomputer, or cloud. Dask also provides specialized data structures to aid in this.

Multiprocessing manages tasks only within a single computer. It may be easier to use than Dask for some tasks, but is less flexible and less scalable. If you're using Dask, you probably don't want multiprocessing. The two will interfere with each other.

Numba is a just-in-time compiler for Python that can provide significant acceleration to code. If most of your work is in Numpy, this is not likely to offer much advantage because Numpy is already highly-optimized. If you're using Python for loops or other high-level, interpreted language features to do the bulk of your computation, then Numba is good for you.

joblib has a lot of overlap with Dask, though you might benefit from its memoization features if you use a lot of recursion.

PyOpenCL offloads array computation to a GPU. This can probably be used in conjunction with Dask and Numba; however, you likely have only one GPU per machine so using PyOpenCL indiscriminately will create contention for that GPU and, essentially, limit you to only a few processes per node.

  • 3
    $\begingroup$ It might be worth being even more explicit that using numpy actually means working with its vectorized function calls, etc. I've found some difficulty in demonstrating to students that just using numpy data structures won't magically speed things up. $\endgroup$ – origimbo Feb 6 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.