I have been granted access to a cluster running PBS and I'd like to run a Evolutionary Algorithm (EA) on it.
To those unfamiliar with EAs, Wikipedia summarizes it as:
Part One: Generate the initial population of individuals randomly (first generation).
Part Two: Repeat the following steps until termination:
- Evaluate the fitness of each individual in the population
- Select the fittest individuals for reproduction (parents)
- Breed new individuals through crossover and mutation operations to give birth to offspring.
- Replace the least-fit individuals of the population with new individuals.
Evaluating the fitness of a individual may take up to 1 hour, and that is running on a node with specialized hardware (Xeon Phi). Everything else is relatively cheap (can be executed in less than 60 seconds on a single CPU).
What I planned to do was:
Generate the initial population on my own computer and upload it to the cluster, then launch a job "ea_loop" that:
- Launches N jobs, with N=number on individuals in the population, to evaluate the fitness of the individuals.
- Every 60 seconds check if all of jobs finished.
- When all jobs have finished, execute steps 2, 3 and 4 of the EAs second part.
- Check if a termination criteria has been met, if not goto 1.
My main problem is: it appears I can not launch the subjobs, I get the following error message when I try to launch them.
qsub: Bad UID for job execution
I Google about this, but pretty much all hits were about job dependencies, which I don't think is an adequate solution. The easy way out would be to run the "ea_loop" in the headnode, but computing on the head node is frowned upo (and probably disallowed). How should I go about this?
My secondary problems are related to the inelegance of my planned solution. Checking if the jobs finished every 60s seconds is somewhat smelly; is there a way to use some sort of callback?