# Parallelizing a for-loop in Python

Are there any tools in Python that are like Matlab's parfor? I found this thread, but it's four years old. I thought maybe someone here might have more recent experience.

Here's an example of the type of thing I'd like to parallelize:

X = np.random.normal(size=(10, 3))
F = np.zeros((10, ))
for i in range(10):
F[i] = my_function(X[i,:])


where my_function takes an ndarray of size (1,3) and returns a scalar.

At the least, I'd like to use multiple cores simultaneously---like parfor. In other words, assume a shared memory system with 8-to-16 cores.

• Lots of results on google. These seem pretty simple: blog.dominodatalab.com/simple-parallelization quora.com/What-is-the-Python-equivalent-of-MATLABs-parfor – Doug Lipinski May 7 '15 at 0:28
• Thanks, @doug-lipinski. Those examples, like others I've found while googling, have some trivial computation based on the iteration index. And they always claim the code is "incredibly easy." My example defines the arrays (allocates the memory) outside the for-loop. I'm okay doing it some other way; that's just how I do it in Matlab. The tricky part that seems to buck those examples is getting part of a given array to the function inside the loop. – Paul G. Constantine May 7 '15 at 0:46

Joblib does what you want. The basic usage pattern is:

from joblib import Parallel, delayed

def myfun(arg):
do_stuff
return result

map(delayed(myfun), arg_instances))


where arg_instances is list of values for which myfun is computed in parallel. The main restriction is that myfun must be a toplevel function. The backend parameter can be either "threading" or "multiprocessing".

You can pass additional common parameters to the parallelized function. The body of myfun can also refer to initialized global variables, the values which will be available to the children.

Args and results can be pretty much anything with the threading backend but results need to be serializable with the multiprocessing backend.

Dask also offers similar functionality. It might be preferable if you are working with out of core data or you are trying to parallelize more complex computations.

• I see zero value added to use the battery including multiprocessing. I woukd bet that joblib is using it under the hood. – Xavier Combelle May 19 '17 at 18:13
• It has to be mentioned that joblib is not magic, the threading backend suffers from the GIL bottleneck and the multiprocessing backend brings large overhead due to the serialization of all parameters and return values. See this answer for the low-level details of parallel processing in Python. – Jakub Klinkovský May 27 '17 at 14:49
• I can't find a combination of function complexity and number of iterations for which joblib would be faster than a for-loop. For me, it has the same speed if n_jobs=1, and is much slower in all other cases – Aleksejs Fomins Jan 7 '19 at 11:15
• @AleksejsFomins Thread based parallelism will not help for code that does not release the GIL but a significant number do, particularly data science or numerical libraries. Otherwise you need mutiprocessing, Jobli supports both. The multiprocessing module now also has parallel map that you can use directly. Also if you use mkl compiled numpy it will parallelize vectorized operations automatically without you doing anything. The numpy in Ananconda is mkl enabled by default. There is no universal solution though. Joblib is very low fuss and there were fewer otions in 2015. – Daniel Mahler Jan 12 '19 at 9:13
• Thanks for your advice. I remember trying multiprocessing before and even writing a few posts, because it did not scale as I expected. Maybe I should give it another look – Aleksejs Fomins Jan 12 '19 at 11:43

What you're looking for is Numba, which can auto parallelize a for loop. From their documentation

from numba import jit, prange

@jit
def parallel_sum(A):
sum = 0.0
for i in prange(A.shape[0]):
sum += A[i]

return sum


Without assuming something special on my_function choosing multiprocessing.Pool().map() is a good guess for parallelizing such simple loops. joblib, dask, mpi computations or numba like proposed in other answers looks not bringing any advantage for such use cases and add useless dependencies (to sum up they are overkill). Using threading as proposed in another answer is unlikely to be a good solution, because you have to be intimate to the GIL interaction of your code or your code should do mainly input/output.

That said numba might be a good idea to speed up sequential pure python code, but I feel this is outside of the scope of the question.

import multiprocessing
import numpy as np

if __name__ == "__main__":
#the previous line is necessary under windows to not execute
# main module on each child under windows

X = np.random.normal(size=(10, 3))
F = np.zeros((10, ))

pool = multiprocessing.Pool(processes=16)
# if number of processes is not specified, it uses the number of core
F[:] = pool.map(my_function, (X[i,:] for i in range(10)) )


There is some caveats however (but which should not impact most of applications):

• under windows there is no fork support, so an interpreter with main module is launched at the startup of each child, so it might have an overhead (ad it is the reason for the if __name__ == "__main__"
• The arguments and the results of my_function are pickled and unpickled, it might be a too big overhead, see this answer for reducing it https://stackoverflow.com/a/37072511/128629 . It also makes non pickable objects unusables
• my_function should not depend on shared states like communicating with global variables because states are not shared between process. pure functions (functions in the mathematical senses) are example of functions which not share states

My impression of parfor is that MATLAB is encapsulating implementation details, so it could be using both shared memory parallelism (which is what you want) and distributed memory parallelism (if you're running a MATLAB distributed computing server).

If you want shared memory parallelism, and you're executing some sort of task parallel loop, the multiprocessing standard library package is probably what you want, maybe with a nice front-end, like joblib, as mentioned in Doug's post. The standard library isn't going to go away, and it's maintained, so it's low-risk.

There are other options out there, too, like Parallel Python and IPython's parallel capabilities. A quick glimpse at Parallel Python makes me think that it's closer to the spirit of parfor, in that the library encapsulates details for the distributed case, but the cost of doing so is that you have to adopt their ecosystem. The cost of using IPython is similar; you have to adopt the IPython way of doing things, which may or may not be worth it to you.

If you care about distributed memory, I recommend mpi4py. Lisandro Dalcin does great work, and mpi4py is used in the PETSc Python wrappers, so I don't think it's going away anytime soon. Like multiprocessing, it's a low(er)-level interface to parallelism than parfor, but one that is likely to last for a while.

• Thanks, @Geoff. Do you have any experience working with these libraries? Maybe I'll try using mpi4py on a shared memory machine / multicore processor. – Paul G. Constantine May 7 '15 at 2:18
• @PaulGConstantine I've used mpi4py successfully; it's pretty painless, if you're familiar with MPI. I have not used multiprocessing, but I have recommended it to colleagues, who said it worked well for them. I've used IPython, too, but not the parallelism features, so I can't speak to how well it works. – Geoff Oxberry May 7 '15 at 2:23
• Aron has a nice mpi4py tutorial he prepared for the PyHPC course at Supercomputing: github.com/pyHPC/pyhpc-tutorial – Matt Knepley May 17 '15 at 2:17

Before looking for a "black box" tool, that can be used to execute in parallel "generic" python functions, I would suggest to analyse how my_function() can be parallelised by hand.

First, compare execution time of my_function(v) to python for loop overhead: [C]Python for loops are pretty slow, so time spent in my_function() could be negligible.

>>> timeit.timeit('pass', number=1000000)
0.01692986488342285
>>> timeit.timeit('for i in range(10): pass', number=1000000)
0.47521495819091797
>>> timeit.timeit('for i in xrange(10): pass', number=1000000)
0.42337894439697266


Second check if there is a simple vector implementation of my_function(v) that does not require loops: F[:] = my_vector_function(X)

(These two first point are pretty trivial, forgive me if I mentioned them here just for completeness.)

Third and most important point, at least for CPython implementations, is to check whether my_function spends most of it's time inside or outside the global interpreter lock, or GIL. If time is spent outside the GIL, then the threading standard library module should be used. (Here an example). BTW, one could think of writing my_function() as a C extension just to release the GIL.

Finally, if my_function() does not release the GIL, one could use the multiprocessing module.

You can try Julia. It's pretty close to Python, and has a lot of MATLAB constructs. The translation here is:

F = @parallel (vcat) for i in 1:10
my_function(randn(3))
end


This makes the random numbers in parallel too, and just concatenates the results in the end during the reduction. That uses multiprocessing (so you need to do addprocs(N) to add processes before using, and this is also works on multiple nodes on an HPC as shown in this blog post).

You can also use pmap instead:

F = pmap((i)->my_function(randn(3)),1:10)


If you want thread parallelism, you can use Threads.@threads (though make sure you make the algorithm thread-safe). Before opening Julia, set the environment variable JULIA_NUM_THREADS, then it's:

Ftmp = [Float64[] for i in Threads.nthreads()]