# Python: Multistart optimization using parallel programming

[I previously asked my question on StackOverflow but this site may be more appropriate]

In Matlab, I am currently using the MultiStart as an optimization algo in a parallel setup for a computer cluster. For example, this is my Matlab code:

opts = optimoptions(@fmincon,'Algorithm','sqp','Use Parallel','Always'); %The options for the algo. The key here is the Use Parallel
problem = createOptimProblem('fmincon','objective',...
@(x) x.^2 + 4*sin(5*x),'x0',3,'lb',-5,'ub',5,'options',opts); %x is the variable I wish to min/max
ms = MultiStart;
[x,f] = run(ms,problem,20) %20 stands for the number of initial random variables


This code works very well when I submit a batch job on the cluster for X processors. I am looking online for a similar algo that I could implement in parallel using Python and I am bit loss. Any similar algo is available for Python?

UPDATE As for my cluster requirements, multithreading is not acceptable. Usually I use mpi4py.

• Have you looked at Coopr (which includes Pyomo)? – Christian Clason Jul 2 '14 at 17:14
• Interesting ... not so trival to see how to implement a MultiSearch from their help guide but I will look into this more. Thanks – Plug4 Jul 2 '14 at 19:18

From what I can tell, MultiStart is not an optimization algorithm per se, but a framework that runs a given optimization algorithm (in your code, SQP via fmincon) in parallel for a set of random starting points (i.e., a separate, independent, instance of fmincon for each starting point). Since the instances do not need to communicate at all, this does not require MPI (the technical term is "embarrassingly parallel"), so you can use any job server to launch the instances, each on a separate node.
Since Python is (if you so choose) much more low-level than Matlab, you can implement this yourself rather easily: Pick an implementation of a minimization algorithm (the closest thing to fmincon would be SciPy's scipy.optimize.minimize) and wrap it in a multiprocessing Pool; the module joblib seems to provide a convenient wrapper for this. If you run this on a cluster, you could try dispy.
• yes you are absolutely right with your first point about MultiSearch not being an optimizaiton algo per se. Putting parallel prog. aside, can scipy.optimize.minimize be set to start with random starting points? Also, correct me if I am wrong, but multiprocessing (I was told) will not work on cluster and I usually use mpi4py to parallel job on a cluster. Is Coopr my only alternative to run scipy.optimize.minimize parallel on a cluster? – Plug4 Jul 2 '14 at 19:54
• What I have in mind is if scipy.optimize.minimize can take multiple random starting values like in Matlab's Multisearch, then I augment the performance with mpi4py ... no idea if this is feasible. – Plug4 Jul 2 '14 at 19:56
• No, minimize takes only a single starting point. What you would do is write a loop, where in each iteration you select a random starting point and run a new instance of minimize on that and collect the results. Then you only need a framework that allows you to execute the loop in parallel. If multithreading is not acceptable, simply replace it with a more suitable job server for Python. The best idea is to ask your cluster admin what they support. – Christian Clason Jul 2 '14 at 20:05