I'm not very familiar with PBS (we have torque installed here), and I have only used it to run one process per job so far, so bear with me.
The actual problem
I am trying to use Mathematica on a cluster. Mathematica's parallel processing is implemented by launching multiple worker processes (called subkernels), and sending commands to them to be run.
Mathematica has some support for PBS, and is able to launch the subkernels as PBS jobs. It does this using the
While it is the subkernels that do most of the work (actually I have some C code linked into them), it is the master Mathematica kernel that collects the results. Therefore the master kernel also uses a small but nonzero amount of CPU time. This means that I cannot launch the master kernel on the headnode because after it has used up a certain amount of CPU-time, it will be auto-terminated (manual launching on the headnode is not allowed, only through
qsub). So I would need to launch the master kernel using
qsub as well. This will result in a job launching more jobs (
qsub being called from within a job submitted with
qsub), which apparently does not work.
It is possible to easily test this like this:
Launch an "interactive job" with
qsub -I (this is actually how I'd prefer to launch the master kernel so I can manually monitor the progress of the calculation), then in the shell that you get try to submit another job with
qsub: these nested
qsubs result in a
qsub: Job rejected by all possible destinations error.
Question 1: Is there a way to get nested
qsubs to work? Can a job queue up more jobs? This would be the ideal solution.
Question 2: If this is not allowed, then what is the correct way to launch more than one process from a job, and have them distributed on free cluster nodes? I assume I need to go through PBS in some way, ask it which machines are free, start the processes on those machines (how? with
ssh executablename?), then tell PBS how many processes I've actually started on which machine. Is this correct? If yes, how can the details be implemented?
I'd prefer not to have to go this second route because it would involve re-implementing Mathematica's subkernel launching with a different mechanism then
qsub. It is a lot of extra work.