0
$\begingroup$

I don't know if anyone is using SGE anymore (what is the world of HPC using nowadays?), but I need to work on version 6.2u3, I believe, from sometime back in '11. I'm using it on a cluster of 64 nodes, 12 cores each.

Thing is, I'm not sure what requesting X cores really means. Is it purely advisory? I've tried using both SMP and OpenMPI environments, and the impression I get is that spawning more threads within the application always increases performance, irrespective of how many cores you request. Even if you request a single core using -pe smp 1 or -pe openmpi 1, you get an almost doubling in performance going from 1 to 2 to 4 threads.

What then, is the purpose of scheduling cores?

Another question is related: I'm getting a doubling of performance in going from 1 to 2 to 4 threads. However, at about 8 or 10 threads (i.e. supposedly, 8 or 10 cores) the linear relationship breaks down, and in fact at 12 threads computation takes longer than that at 8 threads. Even using -pe smp 12. I've tried this with a simple, embarrassingly parallel problem (generating an array of numbers), so there should be no contention. And I don't think it's thread overhead, because the workload is quite large, taking a few seconds.

Looking at the nodes/cores scheduled, I'm told by the system that I have 12 cores on the same node (usually, but not always), yet still performance decreases at 12 threads. I suspect that SGE does not prevent third-party work from running on that node, even though I've requested enough cores to have 1 core per thread, and this is leading to thread interference.

Am I on the right track or have I missed the point entirely?

Thanks.

Improve this question
$\endgroup$
2
  • $\begingroup$ I suspect the number of people who still know the details of the Sun Grid Engine is rather close to zero these days... $\endgroup$
    – Wolfgang Bangerth
    May 25, 2017 at 16:55
  • 1
    $\begingroup$ Gotta admit that cracked me up a bit. I'm forced to use it at my university, though :) $\endgroup$
    – Kristian D'Amato
    May 25, 2017 at 16:57

1 Answer 1

Reset to default
1
$\begingroup$

SGE has been used on the Lonestar4 supercomputer (Texas Advanced Computing Center). Lonestar4 has been decommissioned in 2015; however, a very extensive Lonestar4 user guide has been preserved on TACC website archive.

The SGE: Running subsection of this guide provides a pretty detailed description of the variables and recipes on setting them up.

Regarding your question, you should look into "wayness" and the command will have a syntaxis:

-pe <TpN>way <NoN x NumCoresPerNode>

Here, NumCoresPerNode - number of cored on the nodes of your cluster, NoN - number of nodes, TpN - tasks per node.

Say, you have a computing cluster consisting of nodes with two 6-Core processors. Then,

-pe 12 way 48

requests 4 nodes with 12 tasks per node.

Regarding, if the whole node is blocked if you requested only part of its cores, that probably depends on the configuration of the SGE. On Lonestar4, the node was blocked and reserved only for you even if you requested only 1 core out of the 12 available.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.