1
$\begingroup$

I have a 32 cores machine, and I need to run 16 different dynamics simulations in parallel on it. I want the 16 jobs to run in parallel, not sequentially, on the same machine. The 16 dynamics input are in different folders and they requires the same time to complete.

pbs script:

#!/bin/bash -l
#PBS -l nodes=...:ppn=32,walltime=...
#
#PBS -N dynamics

export OMP_NUM_THREADS=32

export PATH=mypath:$PATH

export DIR="my folder"

export WORK=/scratch/Work/

cp -r $DIR/dyn1 $WORK/.

cp -r $DIR/dyn2 $WORK/.

...

cp -r $DIR/dyn3 $WORK/.

cd $WORK

my_dynamic_program1.x 
my_dynamic_program2.x
$\endgroup$
  • $\begingroup$ What is the reason that they must run in parallel? If there is a condition such, the last program has to run on the output of all others, this can be specified with job depdendencies. So you would submit 16 independent jobs and the last one has the condition to only start after the 15 other have been completed. $\endgroup$ – Bort Apr 4 at 9:38
  • $\begingroup$ I can't speak for PBS, but in SLURM you could ostensibly use a job-array to create appropriate folders (via job script), launch the simulation binding it to a CPU core and restricting it to a certain node. You'd have 1 batch script generating 16 jobs and the scheduler should launch them all on that machine. I guess there's something similar for PBS. $\endgroup$ – Nox Apr 4 at 11:08
  • $\begingroup$ @Bort: Could you better explain why my answer wouldn't work for you? $\endgroup$ – Richard May 4 at 16:04
0
$\begingroup$

There are a few options, but these two come to mind as the most straight-forward:

You could submit 16 different jobs to your scheduler.

You could modify your script to start your programs in the background:

my_dynamic_program1.x &
my_dynamic_program2.x &

(the & says: start running this command and immediately run the next one as well)

You could even do:

cd dir1
./prog1.exe &
cd dir2
./prog2.exe &
...
$\endgroup$
  • $\begingroup$ Thank your for your answer. The problem is that I need to start more calculations at the same time on one single node so I tried with mpirun but doesn't work (mpirun -np 32 ./dynamic_program1-16.x &). $\endgroup$ – Fra Apr 4 at 10:05
  • $\begingroup$ More than what? His answer seems to answer your question as stated. $\endgroup$ – Victor Eijkhout Apr 4 at 14:51
0
$\begingroup$

If you system does not have a job launcher facility (or if PBS does not have that built in) you can do a couple of things.

  • Start all 16 processes, and hope that the OS will nicely spread them over the cores.
  • Start all 16 processes, but prefix them with numactl to fix them to a certain core.
  • Use mpirun but with the following script:

mpi -np 16 parallel_script

where your script:

#!/bin/bash
if [ $PMI_RANK -eq 0 ] ; then 
  ./program1
else if [ $PMI_RANK -eq 1 ] ; then
  ./program2
et cetera.

The variable PMI_RANK may be system-dependent.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.