Low performance on sge cluster

I'm having an issue with my project. When I run the code of Monte-Carlo simulation, on the local server (the machine in my office) it runs on a rate of roughly 100000 steps per 24 hours. When I run it on the cluster the rate drops to about half of that or even less.

I am trying to figure this but I can't find any reason. I don't want you to read my code and analyze it mainly because this is not your work. I only ask for you experience for any suggestion. What I got so far is this:

1. I don't to any parallel computing.
2. When I run the code on the local machine, I don't usually run anything heavy aside from it.
3. I do run, several copies of the code on the local machine but not more than the number of processor - 1 on that machine. The IT guys aren't sure, but they think that no dual processing are enabled on the local server. (There was an unsolved issue with that)
4. When I run a single copy of the code on the cluster it runs okay.
5. A significant drop of performance occur only when the number of copies is 8.
6. There are two kinds of machines on the cluster. 16 nodes (8 dual processor) and 24 nodes (I think that those are 6 qudriple processors).
7. The code is written in C++. Even though it is considered bad programming, all my variables are global and I access them directly (not via function).
8. I compile the code using icc -O3 -fast -ip. Removing any of those flag either reduced the performance or didn't have any noticeable affect.

I think, from all of this that this is an issue with memory management but I don't know how to test this.

• Did any of encountered something similar?
• How can I test if this is a memory issue?
• Is there another flag I could add to my compilation?
• Are there any tips for efficient memory usage in c++?

Thank you for any help

• Can you get usage reports back from your cluster? How many people use the cluster? This sounds like the resources devoted to your jobs are less than those of your local workstation, which for single jobs, or heavily used clusters, wouldn't be surprising. Jan 20 '12 at 20:13
• What are the clock rates of your local server and the cluster nodes? What kind of processors are they, specifically? Most processors can do more than one operation per clock cycle, but perhaps your cluster's processors cannot. Jan 20 '12 at 20:15
• If your code is limited by memory bandwidth, adding more cores may not help. Check the process affinity and run some tests where you put only one job per socket versus packing the jobs together. Jan 20 '12 at 20:20

• Thanks. Do you happen to know if type_a->type_b->type_c is more expensive than direct access to type_c? (c++) Jan 21 '12 at 6:36