Tell me more ×
Computational Science Stack Exchange is a question and answer site for scientists using computers to solve scientific problems. It's 100% free, no registration required.
up vote 1 down vote favorite

I am trying to figure out how to write a proper job file for the Sun Grid Engine so I can run a parallel computation using Mathematica on multiple nodes. Unfortunately almost all tutorials on the topic assume using MPI, which is not the case here.

In order to get this working, I need to find the answer to a few very specific questions:

  1. When I request multiple nodes, will the job file be run on all of them, or only on a single node? I need to run it only on a single node in this case, and start a single process which we can call controlling process.

  2. How can I access the names / IP addresses of all the other nodes? I think I have the answer to this one: the PE_HOSTFILE variable should point to a file that lists the names of all nodes.

  3. How can I let the controlling process start arbitrary processes at arbitrary times on any node that I can use? Should a simple ssh nodename command work for this, or would this require a password in a typical setup?

share|improve this question

1 Answer

up vote 4 down vote accepted
  1. The job script is run only on a single node by SGE. It is run on the first host listed in PE_HOSTFILE
  2. PE_HOSTFILEcan be parsed to get the names of all the nodes
  3. Using SSH without a passphrase internal to a cluster is a common setup, but it depends on who setup the machine you'll be using. I would ask them.

In addition, some HPC sites supply tools that might let you launch these tasks in a simpler manner. You should check with the site where you will be running.

share|improve this answer
If ssh without a passphrase is not set up, which is the next most likely to work alternative for point 3.? – Szabolcs Feb 12 at 17:38
SSH without passphrase can be set up by you, actually. However, you can easily test it from your job script. Just pick off the second host from PE_HOSTFILE and try ssh $host uptime (or similar). If it doesn't work, you can setup your own passphraseless SSH. – Bill Barth Feb 12 at 17:52
Thanks for the advice, I can do that. – Szabolcs Feb 12 at 17:55
Unfortunately public key authentication seems to be disabled :-( – Szabolcs Feb 12 at 18:52
Did you try it within a job, or from outside? – Bill Barth Feb 12 at 19:06
show 4 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.