# What does Python offer for distributed/parallel/GPU computing?

Using the SciPy/NumPy libraries, Python is a pretty cool and performing platform for scientific computing. I just wonder: When I have to go parallel (multi-thread, multi-core, multi-node, gpu), what does Python offer?

I'm mostly looking for something that is fully compatible with the current NumPy implementation. The coolest thing would be an implementation of NumPy and some other basic functionality (list comprehension for example) that is fully parallel and also can use GPUs. But I'm probably asking for a bit too much ;-)

(It's not that we are unable to switch to solution in C/C++, it's more that we'd really like to leverage the advantages of Python above these languages also in a parallel enivronment.)

• What scientific computing library are you using in C++? Petsc, for instance, has Python bindings, so essentially you get the same functionalities. – Federico Poloni Jun 22 '16 at 14:18

Here's a few options that are relatively easy to work with:

• One node - multiprocessing is the most straightforward thing to do. multiprocessing.map works well for an embarrasingly parallel problem.
• GPU - pyCUDA allows you to do some GPU level programming with just python. I haven't toyed with it that much.
• MPI - mpi4py- Exposes MPI interface at the python level. If you're familiar with MPI, this is useful.

There's a ton of options out there. Here's a page that attempts to catalog some of them.

(I wanted to give more links, but apparently I don't have the rep for that. A quick search on these terms should turn up documentation)

Among the things you can try is pyMPI for MPI support in python. We have a good number of users at our HPC site that use it for parallelism. It's not an automated way to get there through NumPy, but you should look at its features carefully.

IPython has a package for parallel computing called ipyparallel.

An overview from their documentation:

This architecture abstracts out parallelism in a very general way, which enables IPython to support many different styles of parallelism including:

• Single program, multiple data (SPMD) parallelism.
• Multiple program, multiple data (MPMD) parallelism.
• Message passing using MPI.
Most importantly, IPython enables all types of parallel applications to be developed, executed, debugged and monitored interactively. Hence, the I in IPython. The following are some example usage cases for IPython: