I just ran a bash script on a remote computer with a for loop that repeatedly executes the following line of code two hundred times:

mpiexec -n 8 ./my_computationally_expensive_program

For some strange reason, my local computer crashed, and I assumed that my shell command was abruptly interrupted. However, when I logged back into the remote computer, I find that the program was still running in the background. I know this because new files are constantly being generated without any program on the current shell. Also, I did not use PBS or any other job scheduler. I simply ran the script from the command line. Any help would be greatly appreciated! :)

  • 1
    $\begingroup$ This is better off at Unix.SE. But, last time I encountered this, I init 6ed the computer since I knew the su password. This restarted the computer. Fortunately, no one else was using it. $\endgroup$ – Inquest Mar 21 '12 at 14:01
  • $\begingroup$ If I had superuser priviledges, I would definitely use them :) Alas, I can't :) $\endgroup$ – Paul Mar 21 '12 at 14:03
  • 1
    $\begingroup$ I know this is late but pidof my_computationally_expensive_program followed by kill on the offending pid's. $\endgroup$ – Godric Seer Mar 26 '14 at 19:17

One each node run the command

ps -u your_name | grep prog_name | awk '{print $1}' | xargs kill -KILL

To automate you can write a small bash script/command that reads the hostnames listed in machinefile and uses ssh/rsh to execute the above. Alternatively you can use pkill -u user if you want to kill all jobs associated with the user running on the node.

Remember you can run commands via ssh/rsh. For example

ssh hostname 'ls -l'

  • $\begingroup$ I tried running the program on one shell and running you suggestion on a separate shell. Though all the pid's were removed, the program did not seem to come to a hault. $\endgroup$ – Paul Mar 21 '12 at 14:31

This might or might not work. However, this works on my laptop if I want to prevent the execution of a certain program. However, this is a really bad infinite loop type answer but it might work.

while true;do
killall name
sleep 5s

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.