I am using an HPC cluster to launch parallel simulations. At the moment, there are 48 free cores over 4 nodes but not each node do not have the same number of free cores. The distribution is something like this :

node1: 8 cores free

node2: 8 cores free

node3: 8 cores free

node4: 24 cores free

I want to utilize all of these resources to launch a simulation. I am using PBS as the job scheduler and normally I assign resources using qsub -l nodes=x:ppn=xxxx

How can I assign resources in this case?

  • 2
    $\begingroup$ You should commit your job with your needed resources in mind and not what is available. Let the scheduler figure out how to distribute and launch jobs since you are probably not alone on the cluster. $\endgroup$
    – Bort
    May 18 '18 at 9:31
  • $\begingroup$ I understand your point. But such a situation at my HPC is quite often.. There is another queue where the nodes have only 16cores. and job on this queue get promoted to nodes with 24cores. Therefore, 1/3 of computational power of these cores is not used. $\endgroup$ May 18 '18 at 10:02
  • 2
    $\begingroup$ As a user, the choices made by the HPC admins are not your fault. If you've set the total processor count (procs) or node count nodes and minimum processors per node (ppn) to what your code requires to run, and are matching any additional criteria on job size your cluster requested of you, you're doing your job. $\endgroup$
    – origimbo
    May 18 '18 at 10:28

When you write/launch a PBS script, you are not supposed to know what resources are available, you are supposed to describe what resources do you need. And while the efficient use of HPC resources is important, you should not take the work of the job scheduler.

However, I guess you are interested in the ability to request different ppn for different nodes. This can be done using the following type of request (with +):

#PBS -l nodes=1:ppn=3+1:ppn=8+24
#PBS -l naccesspolicy=SINGLEJOB -n

That is supposed to request 3 nodes with 8 ppn and 1 node with 24 ppn. The question is if your scheduler is configured to process such requests is open. More about that here or in the official PBS manual.

Now, since the resource availability is changing and dynamic with time, your PBS scripts have to be generated using some kind of a shell script that will check the available resources and generate the corresponding PBS file.

As a summary: I, personally, do not advise going this route and certainly check with HPC cluster administrators if such dynamic resource requests are even compliant with "good citizen" policy. And even if they are, I still think that you are taking a chunk of the scheduler work into your hands unnecessarily.

  • $\begingroup$ Thank you for the answer.. #PBS -l nodes=1:ppn=24+3:ppn=8 worked smoothly for my case! .. $\endgroup$ May 20 '18 at 13:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.