2
$\begingroup$

I apologize if this is a relative simple question. I am very new to setting up a cluster computer and I am attempting to make heads and tails of the information.

I was reading on setting up a cluster with MPICH in ubuntu. And it seemed like it was a "simple" process. The process was a simple 1,2,3 step process.

Then I came upon process managers such as openMosix. From the steps outlines in the MPICH (https://help.ubuntu.com/community/MpichCluster) it seemed like I don't need to setup a process manangmenet but at the same time, these programs help to manage parallel jobs. And MPICH has an extension to talk to the process manager.

So, when would I use a process manager? Would this be on large clusters? Or should I always install process managers and they are used in conjunction with MPI? What is the difference between an MPI (whether it be MPICH or OpenMPI) and the process manager? Or does the process manager utilize MPI?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You should use some sort of process or resource management (aka a scheduler) once more than one person at a time will try to use the cluster. Unless you have some other way to guarantee that person A won't use person B's nodes, you need a scheduler. Sometimes, in some small labs, this is done by agreement of two people who literally sit next to each other, but even that is bound to fail, so installing an open source scheduler is the route to go. This isn't for the faint of heart, but it's also not terribly hard. There are lots of documents out there to help you.

Improve this answer
$\endgroup$
6
  • $\begingroup$ ok, that makes sense. So it would be good if for example I have 5 different programs that need the cluster? $\endgroup$
    – philm
    Jul 5, 2016 at 22:24
  • $\begingroup$ @philm, if only one person is running those programs, then they can cut the host list up into 5 distinct chunks for each program and just run them without a scheduler, but if 2 or more people need access, then a scheduler is a near must. $\endgroup$
    – Bill Barth
    Jul 5, 2016 at 22:26
  • $\begingroup$ Oh ok, so it is like if there are two different accounts running 1 MPI program each vesus 1 account running 2 different MPI programs? $\endgroup$
    – philm
    Jul 5, 2016 at 22:46
  • $\begingroup$ @philm, sure. I suppose. If the multiple people I mentioned are using the same account, that's a recipe for 6 different kinds of trouble and should be avoided for security and sanity reasons at the very least. $\endgroup$
    – Bill Barth
    Jul 5, 2016 at 22:47
  • 1
    $\begingroup$ @philm, yes, even in the case of one person, if the cluster is 10 nodes and that person needs to run a 6-node and a 5-node job of unknown length, they can submit them both, and the 5-node job will wait until the 6-node job finishes. Then the 5-node job will run almost immediately leaving virtually no idle time on 5 of the nodes. This is better than the user having to check the status of the first job constantly to see when to run the next one. $\endgroup$
    – Bill Barth
    Jul 5, 2016 at 23:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.