Parallelizing a for-loop in Python

Question

Are there any tools in Python that are like Matlab's parfor? I found this thread, but it's four years old. I thought maybe someone here might have more recent experience.

Here's an example of the type of thing I'd like to parallelize:

X = np.random.normal(size=(10, 3))
F = np.zeros((10, ))
for i in range(10):
    F[i] = my_function(X[i,:])

where my_function takes an ndarray of size (1,3) and returns a scalar.

At the least, I'd like to use multiple cores simultaneously---like parfor. In other words, assume a shared memory system with 8-to-16 cores.

Answer 1

Joblib does what you want. The basic usage pattern is:

from joblib import Parallel, delayed

def myfun(arg):
     do_stuff
     return result

results = Parallel(n_jobs=-1, verbose=verbosity_level, backend="threading")(
             map(delayed(myfun), arg_instances))

where arg_instances is list of values for which myfun is computed in parallel. The main restriction is that myfun must be a toplevel function. The backend parameter can be either "threading" or "multiprocessing".

You can pass additional common parameters to the parallelized function. The body of myfun can also refer to initialized global variables, the values which will be available to the children.

Args and results can be pretty much anything with the threading backend but results need to be serializable with the multiprocessing backend.

---

Dask also offers similar functionality. It might be preferable if you are working with out of core data or you are trying to parallelize more complex computations.

Answer 2

What you're looking for is Numba, which can auto parallelize a for loop. From their documentation

from numba import jit, prange

@jit
def parallel_sum(A):
    sum = 0.0
    for i in prange(A.shape[0]):
        sum += A[i]

    return sum

Answer 3

Without assuming something special on my_function choosing multiprocessing.Pool().map() is a good guess for parallelizing such simple loops. joblib, dask, mpi computations or numba like proposed in other answers looks not bringing any advantage for such use cases and add useless dependencies (to sum up they are overkill). Using threading as proposed in another answer is unlikely to be a good solution, because you have to be intimate to the GIL interaction of your code or your code should do mainly input/output.

That said numba might be a good idea to speed up sequential pure python code, but I feel this is outside of the scope of the question.

import multiprocessing
import numpy as np

if __name__ == "__main__":
   #the previous line is necessary under windows to not execute 
   # main module on each child under windows

   X = np.random.normal(size=(10, 3))
   F = np.zeros((10, ))

   pool = multiprocessing.Pool(processes=16)
   # if number of processes is not specified, it uses the number of core
   F[:] = pool.map(my_function, (X[i,:] for i in range(10)) )

There is some caveats however (but which should not impact most of applications):

• under windows there is no fork support, so an interpreter with main module is launched at the startup of each child, so it might have an overhead (ad it is the reason for the if __name__ == "__main__"
• The arguments and the results of my_function are pickled and unpickled, it might be a too big overhead, see this answer for reducing it https://stackoverflow.com/a/37072511/128629 . It also makes non pickable objects unusables
• my_function should not depend on shared states like communicating with global variables because states are not shared between process. pure functions (functions in the mathematical senses) are example of functions which not share states

Answer 4

My impression of parfor is that MATLAB is encapsulating implementation details, so it could be using both shared memory parallelism (which is what you want) and distributed memory parallelism (if you're running a MATLAB distributed computing server).

If you want shared memory parallelism, and you're executing some sort of task parallel loop, the multiprocessing standard library package is probably what you want, maybe with a nice front-end, like joblib, as mentioned in Doug's post. The standard library isn't going to go away, and it's maintained, so it's low-risk.

There are other options out there, too, like Parallel Python and IPython's parallel capabilities. A quick glimpse at Parallel Python makes me think that it's closer to the spirit of parfor, in that the library encapsulates details for the distributed case, but the cost of doing so is that you have to adopt their ecosystem. The cost of using IPython is similar; you have to adopt the IPython way of doing things, which may or may not be worth it to you.

If you care about distributed memory, I recommend mpi4py. Lisandro Dalcin does great work, and mpi4py is used in the PETSc Python wrappers, so I don't think it's going away anytime soon. Like multiprocessing, it's a low(er)-level interface to parallelism than parfor, but one that is likely to last for a while.

Answer 5

Before looking for a "black box" tool, that can be used to execute in parallel "generic" python functions, I would suggest to analyse how my_function() can be parallelised by hand.

First, compare execution time of my_function(v) to python for loop overhead: [C]Python for loops are pretty slow, so time spent in my_function() could be negligible.

>>> timeit.timeit('pass', number=1000000)
0.01692986488342285
>>> timeit.timeit('for i in range(10): pass', number=1000000)
0.47521495819091797
>>> timeit.timeit('for i in xrange(10): pass', number=1000000)
0.42337894439697266

Second check if there is a simple vector implementation of my_function(v) that does not require loops: F[:] = my_vector_function(X)

(These two first point are pretty trivial, forgive me if I mentioned them here just for completeness.)

Third and most important point, at least for CPython implementations, is to check whether my_function spends most of it's time inside or outside the global interpreter lock, or GIL. If time is spent outside the GIL, then the threading standard library module should be used. (Here an example). BTW, one could think of writing my_function() as a C extension just to release the GIL.

Finally, if my_function() does not release the GIL, one could use the multiprocessing module.

References: Python docs on Concurrent Execution, and numpy/scipy intro on parallel processing.

Answer 6

You can try Julia. It's pretty close to Python, and has a lot of MATLAB constructs. The translation here is:

F = @parallel (vcat) for i in 1:10
    my_function(randn(3))
end

This makes the random numbers in parallel too, and just concatenates the results in the end during the reduction. That uses multiprocessing (so you need to do addprocs(N) to add processes before using, and this is also works on multiple nodes on an HPC as shown in this blog post).

You can also use pmap instead:

F = pmap((i)->my_function(randn(3)),1:10)

If you want thread parallelism, you can use Threads.@threads (though make sure you make the algorithm thread-safe). Before opening Julia, set the environment variable JULIA_NUM_THREADS, then it's:

Ftmp = [Float64[] for i in Threads.nthreads()]
Threads.@threads for i in 1:10
    push!(Ftmp[Threads.threadid()],my_function(randn(3)))
end
F = vcat(Ftmp...)

Here I make a separate array for each thread, so that way they don't clash when adding to the array, then just concatenate the arrays afterwards. Threading is pretty new so right now there's just the straight use of threads, but I'm sure threaded reductions and maps will be added just like it was for multiprocessing.

Answer 7

i recommend to use joblib library's Parallel and delayed functions use "tempfile" module to create temp shared memory for huge arrays, the examples and usage can be found in here https://pythonhosted.org/joblib/parallel.html