5
$\begingroup$

We have Maui and Torque on our lab's UNIX cluster. Right now, all jobs are served by FIFO. We'd like to implement a more fair policy, but I have not successfully implemented it. The online documentation was not quite clear, and nothing that I've tried to implement has had any effect (I've altered the config file, then restarted Maui). Any suggestions?

$\endgroup$
1
  • $\begingroup$ Heather, the edit piqued my interest, and while I was poking around, I happened across this web page, which may or may not be useful. Also, welcome to scicomp! $\endgroup$ Feb 3, 2012 at 20:46

2 Answers 2

2
$\begingroup$

So as aeismail has suggested, "fair" is in the eyes of the beholder, and in general, any policy that doesn't let a users job run when they think it should will strike them as manifestly "unfair" and will generate nasty emails to the sysadmin. So don't get too caught up in tweaking parameters; come up with a clear, simple policy, and adjust it only slowly over time as you get data, not reactively in response to complaints.

Anyway, in terms of implementing things in maui, I'd suggest starting with fairshare. It lets you set target amounts of resources for groups over some window of time (we use 2 weeks here - pick a length of time over which, for your usage patterns, peakiness tends to average out), and as they get close to that usage, their priority starts to drop. Note that they can still compute if there are idle cycles, but people who haven't used so much over that period will have their jobs take priority.

Note that part of the art here is in defining groups of users correctly, and in deciding on the original target amounts of resources which are "fair".

$\endgroup$
1
$\begingroup$

What are the features of rational queuing policies? I'd argue that they include:

  • They restrict the total length of individual jobs. On a shared machine, jobs should have finite run times. The allotment can perhaps be more generous on a small group cluster (4-5 days) than on a large shared resource operated by a university, national lab or other such organization (where anywhere from 1-4 days may be the norm).

  • They place limits on the amount of the machine a single user can access when jobs are in the queue. There's nothing worse than a machine hog that thinks "all your core are belong to me" [sic]. Thus, there's typically a requirement that no single user can have access to more than, say, one-fourth or one-third of a machine, so long as there are other jobs in the queue.

Perhaps most importantly, though:

  • Queues take into account the amount of resources recently accessed. This is the hardest part, though, because it's probably the most difficult to keep track of. However, the basic idea here is that somebody who has been running on a fourth of the machine for a month straight should have a substantially lower priority than somebody who is returning to start new jobs for the first time in a few months. It is equally important to notice the recently: what you did six months or a year ago shouldn't matter much with respect to your current usage. Therefore, there should be some "half-life" on this: maybe a few weeks, maybe a few months.

It will take some time and tweaking to find the right balance, but these rules lie at the heart of most policies I've seen that haven't led to grumbling and teeth-gnashing.

$\endgroup$
1
  • $\begingroup$ I think Heather is also asking how to implement the policies you suggest in Maui/Torque. $\endgroup$ Feb 5, 2012 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.