PBS: Using different computational resources on different nodes

Question

I am using an HPC cluster to launch parallel simulations. At the moment, there are 48 free cores over 4 nodes but not each node do not have the same number of free cores. The distribution is something like this :

node1: 8 cores free

node2: 8 cores free

node3: 8 cores free

node4: 24 cores free

I want to utilize all of these resources to launch a simulation. I am using PBS as the job scheduler and normally I assign resources using qsub -l nodes=x:ppn=xxxx

How can I assign resources in this case?

Answer 1

When you write/launch a PBS script, you are not supposed to know what resources are available, you are supposed to describe what resources do you need. And while the efficient use of HPC resources is important, you should not take the work of the job scheduler.

However, I guess you are interested in the ability to request different ppn for different nodes. This can be done using the following type of request (with +):

#PBS -l nodes=1:ppn=3+1:ppn=8+24
#PBS -l naccesspolicy=SINGLEJOB -n

That is supposed to request 3 nodes with 8 ppn and 1 node with 24 ppn. The question is if your scheduler is configured to process such requests is open. More about that here or in the official PBS manual.

Now, since the resource availability is changing and dynamic with time, your PBS scripts have to be generated using some kind of a shell script that will check the available resources and generate the corresponding PBS file.

As a summary: I, personally, do not advise going this route and certainly check with HPC cluster administrators if such dynamic resource requests are even compliant with "good citizen" policy. And even if they are, I still think that you are taking a chunk of the scheduler work into your hands unnecessarily.