11
$\begingroup$

I have experience in coding OpenMP for Shared Memory machines (in both C and FORTRAN) to carry out simple tasks like matrix addition, multiplication etc. (Just to see how it competes with LAPACK). I know OpenMP enough to carry out simple tasks without the need to look at documentation.

Recently, I shifted to Python for my projects and I don't have any experience with Python beyond the absolute basics.

I have 2 questions:

  • Is there a good guide (online PDF) for describing shared memory parallel computing for Python?

  • What is the best way of achieving it? I have seen a bit of ctypes and I'm not sure if that's the best way. (By best I mean one which has a good tradeoff between programmer time and system time. It shouldn't be too tedious to code neither should the execution be slow)

$\endgroup$

2 Answers 2

8
$\begingroup$

[This is my first post and I hope I have not completely misunderstood the use of SE - if so I apologize in advance]

I agree with "bgschaid" that the question is very difficult to answer based on the information provided. It makes a huge difference if you want low-level routines to exploit a multi-core architecture or if you need to exploit parallelism for embarrassingly parallel problems - or something in-between. An overview of different parallel computing possibilities in Python can be found here.

In the former case I for sure recommend to use tools such as NumPy / SciPy which at least in the MKL compiled version from Enthought supports multi-core architectures. Here you can control the number of cores to use via the environment variable "MKL_NUM_THREADS". This relies on highly optimized libraries that we can hardly expect to beat performance wise. I believe it is generally advised to use these high quality and highly optimized libraries whenever possible.

If you wish to exploit parallelism at a coarse level, the Python standard tool multiprocessing is easy to use - and it also supports shared data objects. There are different tools to use as part of the multiprocessing package. I have used map_async (SIMD like) and apply_async (MIMD like) for several problems with good results. The multiprocessing package is quite easy to use and being a standard part of Python means that you can expect other potential users of your code to easily be able to use it. multiprocessing also links to NumPy data objects directly. When using multiprocessing I would recommend you to set the environment variable "MKL_NUM_THREADS" to 1 such that NumPy is only allowed one core for each process/worker - otherwise you could end up in a resource contention between NumPy parallel and multiprocessing which leads to a performance degradation. multiprocessing works fine for a multi-CPU/multi-core architecture under the same operating system. I have used multiprocessing on a shared memory computer with 4 x Xeon E7-4850 CPUs (each 10 cores) and 512 GB memory and it worked extremely well. Shared arrays can be handled by multiprocessing.Array or sharedctypes. You can find the Python documentation here - check the library.pdf file. I have some slides explaining some of the basic parts of this - PM me if you want those.

If you have a cluster configuration with distributed memory I believe mpi4py is likely the preferred tool. I have not used this myself but I know that it is being used a lot in Python parallel programming.

$\endgroup$
3
$\begingroup$

It depends on the level you're trying to program parallel. For instance for matrix/vector stuff the first stop with python would be NumPy/SciPy (they provide an interface to numeric libraries which gives you the full speed of the libraries with the convenience of Python) and from what they write about parallelism it seems that if the libraries are compiled for parallel use then the programs take advantage of multicores for certain operations. (seems like this article is a bit older, os things might have improved in the meantime. Also there are links to other means of parallel programming.

And of course there is mpi4py to directly program MPI (included in the above article)

My bottom line is: if your main interest is vector/matrix-operations and parallelism is something you "only" need to do it fast then you should have a look at the NumPy/SciPy-ecosystem and only if you don't find the stuff there that you need should you consinder writing your own libraries

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.