6
$\begingroup$

I'm not very familiar with PBS (we have torque installed here), and I have only used it to run one process per job so far, so bear with me.

The actual problem

I am trying to use Mathematica on a cluster. Mathematica's parallel processing is implemented by launching multiple worker processes (called subkernels), and sending commands to them to be run.

Mathematica has some support for PBS, and is able to launch the subkernels as PBS jobs. It does this using the qsub command.

While it is the subkernels that do most of the work (actually I have some C code linked into them), it is the master Mathematica kernel that collects the results. Therefore the master kernel also uses a small but nonzero amount of CPU time. This means that I cannot launch the master kernel on the headnode because after it has used up a certain amount of CPU-time, it will be auto-terminated (manual launching on the headnode is not allowed, only through qsub). So I would need to launch the master kernel using qsub as well. This will result in a job launching more jobs (qsub being called from within a job submitted with qsub), which apparently does not work.

It is possible to easily test this like this:

Launch an "interactive job" with qsub -I (this is actually how I'd prefer to launch the master kernel so I can manually monitor the progress of the calculation), then in the shell that you get try to submit another job with qsub: these nested qsubs result in a qsub: Job rejected by all possible destinations error.

Question 1: Is there a way to get nested qsubs to work? Can a job queue up more jobs? This would be the ideal solution.

Question 2: If this is not allowed, then what is the correct way to launch more than one process from a job, and have them distributed on free cluster nodes? I assume I need to go through PBS in some way, ask it which machines are free, start the processes on those machines (how? with ssh executablename?), then tell PBS how many processes I've actually started on which machine. Is this correct? If yes, how can the details be implemented?

I'd prefer not to have to go this second route because it would involve re-implementing Mathematica's subkernel launching with a different mechanism then qsub. It is a lot of extra work.

$\endgroup$
  • $\begingroup$ The simplest solution might be to ask the administrators of the system to remove the restriction on the head node. $\endgroup$ – Dan Jan 30 '12 at 18:47
  • $\begingroup$ @Dan I've already asked and been refused. The reason is that the head-node should not be overloaded, as that'd break the whole queuing system. Are you familiar with PBS? Do you think that what I am asking is impossible or difficult? I have no idea. $\endgroup$ – Szabolcs Jan 30 '12 at 19:12
  • $\begingroup$ I'm pretty familiar with PBS, but I don't know enough about mathematica to give a useful answer. Is it possible to start mathematica processes in the background and then connect to them later? $\endgroup$ – Dan Jan 30 '12 at 20:50
  • $\begingroup$ @Dan That is exactly how it works: the main kernel starts the subkernels, passing them a port number and ip. Then the subkernels attempt to connect to these ports through TCPIP. Practically all I need is launching $n$ processes and distributing them on free nodes. I've never run more than one process from a job file before. $\endgroup$ – Szabolcs Jan 30 '12 at 21:01
  • $\begingroup$ @Szabolics: I was trying to figure out if rcollyer's answer is something that could be done: starting the subkernels and head kernel from the same PBS script and then connecting them later, rather than having the head start the subkernels. $\endgroup$ – Dan Jan 30 '12 at 21:09
3
$\begingroup$

The issue with question 1 is that as separate jobs, they will be unable to communicate with each other since they may run at different times. This method is better for a cluster which you have full control of, and can run semi-interactively on a head node.

Since that is out, the alternative is to launch the kernels yourself from within a script running on the master kernel. To launch the master kernel from a PBS script, either write a mathematica script with the header line

#!/path/to/mathkernel -script

if v.8 is installed on the cluster, or launch the kernel via

/path/to/mathkernel -initfile "file-containing-script"

if v.8 is not installed. In either case, your script needs to open with the following code

Needs["SubKernels`RemoteKernels`"];
kernels = LaunchKernels[ RemoteMachine[#] ]& /@ Environment["PBS_NODEFILE"];

and should close with

CloseKernels[ kernels ];

Then you have you can use the kernels as you wish. The master kernel is already working on one of the nodes, and will be available for processing as the slave kernels finish.

$\endgroup$
1
$\begingroup$

I managed to figure out that nested qsubs do work, but qsub can only be executed on the headnode. Thus one needs to use ssh headnode qsub jobname, where "headnode" is the name of the headnode.

This is still a hack, the better (but lengthier) solution being @rcollyer's suggestion.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.