I don't know if anyone is using SGE anymore (what is the world of HPC using nowadays?), but I need to work on version 6.2u3, I believe, from sometime back in '11. I'm using it on a cluster of 64 nodes, 12 cores each.
Thing is, I'm not sure what requesting X cores really means. Is it purely advisory? I've tried using both SMP and OpenMPI environments, and the impression I get is that spawning more threads within the application always increases performance, irrespective of how many cores you request. Even if you request a single core using
-pe smp 1 or
-pe openmpi 1, you get an almost doubling in performance going from 1 to 2 to 4 threads.
What then, is the purpose of scheduling cores?
Another question is related: I'm getting a doubling of performance in going from 1 to 2 to 4 threads. However, at about 8 or 10 threads (i.e. supposedly, 8 or 10 cores) the linear relationship breaks down, and in fact at 12 threads computation takes longer than that at 8 threads. Even using
-pe smp 12. I've tried this with a simple, embarrassingly parallel problem (generating an array of numbers), so there should be no contention. And I don't think it's thread overhead, because the workload is quite large, taking a few seconds.
Looking at the nodes/cores scheduled, I'm told by the system that I have 12 cores on the same node (usually, but not always), yet still performance decreases at 12 threads. I suspect that SGE does not prevent third-party work from running on that node, even though I've requested enough cores to have 1 core per thread, and this is leading to thread interference.
Am I on the right track or have I missed the point entirely?