What does the priority of a PBS job really mean?

Question

The qsub command which submits PBS jobs has a -p option that allows you to set the priority. From the man page:

Defines the priority of the job. The priority argument must be a integer between -1024 and +1023 inclusive. The default is no priority which is equivalent to a priority of zero.

And that's all that is in the man page. Which leaves me wondering, what does the priority actually mean? For example: does the system just pick out the jobs with the highest priority to run, or does it use some combination of waiting time and priority? Is priority only measured relative to other jobs I've submitted, or does it cross user boundaries? Does a higher number mean higher priority, or is it like Linux's niceness? Basically I'm looking for a less sparse explanation of what this option is good for.

Answer 1

Here is what I know as a user (sysadmins might have a more accurate answer):

• the queue system uses some algorithm to maximize the use of the resources while prioritizing some jobs over others. Let's say there are 100 cores available on the queue. A high priority job (more on this in a second) asks for 128 cores for 24 hours and was submitted first while a low priority job is now asking for 64 cores for 12 hours. If additional cores are not going to get available within 12 hours (thus allowing the big job to run), the queue will let the small job go through the queue even though it has lower priority and was submitted after the other one. It does so because there is no way the bog job will run in the next 12 hours and wants to maximize the use of resources.
• if using a queue where defining priority of a job is not explicitely allowed, different queues (standard, debug, long, special, ...) will give jobs a default priority. This allows more important project to go through the queue faster than less important ones for example. And prevents every computational scientist from setting his/her own job's priority to the max :)
• jobs will also gain priority over time. Not sure if the rate is dependent on the queue, but I believe higher number means higher priority.
• the priority is measured across all jobs in the queue system

Hope this helps !

Answer 2

FrenchKheldar's answer is very good. The details will vary somewhat from system to system, but the basics are all there.

On our system, we have an explanation of the -p flag in particular: it basically is as you suspect like nice, and like nice normal users can only make their jobs have lower (eg, use negative arguments to p) priority than normal. This lets you shuffle around the order of your jobs with respect to each other, but probably doesn't greatly change the ordering of your jobs and someone else's.