2
$\begingroup$

I'm having an issue with my project. When I run the code of Monte-Carlo simulation, on the local server (the machine in my office) it runs on a rate of roughly 100000 steps per 24 hours. When I run it on the cluster the rate drops to about half of that or even less.

I am trying to figure this but I can't find any reason. I don't want you to read my code and analyze it mainly because this is not your work. I only ask for you experience for any suggestion. What I got so far is this:

  1. I don't to any parallel computing.
  2. When I run the code on the local machine, I don't usually run anything heavy aside from it.
  3. I do run, several copies of the code on the local machine but not more than the number of processor - 1 on that machine. The IT guys aren't sure, but they think that no dual processing are enabled on the local server. (There was an unsolved issue with that)
  4. When I run a single copy of the code on the cluster it runs okay.
  5. A significant drop of performance occur only when the number of copies is 8.
  6. There are two kinds of machines on the cluster. 16 nodes (8 dual processor) and 24 nodes (I think that those are 6 qudriple processors).
  7. The code is written in C++. Even though it is considered bad programming, all my variables are global and I access them directly (not via function).
  8. I compile the code using icc -O3 -fast -ip. Removing any of those flag either reduced the performance or didn't have any noticeable affect.

I think, from all of this that this is an issue with memory management but I don't know how to test this.

I have several question about this:

  • Did any of encountered something similar?
  • How can I test if this is a memory issue?
  • Is there another flag I could add to my compilation?
  • Are there any tips for efficient memory usage in c++?

Thank you for any help

Improve this question
$\endgroup$
3
  • $\begingroup$ Can you get usage reports back from your cluster? How many people use the cluster? This sounds like the resources devoted to your jobs are less than those of your local workstation, which for single jobs, or heavily used clusters, wouldn't be surprising. $\endgroup$ – Fomite Jan 20 '12 at 20:13
  • $\begingroup$ What are the clock rates of your local server and the cluster nodes? What kind of processors are they, specifically? Most processors can do more than one operation per clock cycle, but perhaps your cluster's processors cannot. $\endgroup$ – Bill Barth Jan 20 '12 at 20:15
  • $\begingroup$ If your code is limited by memory bandwidth, adding more cores may not help. Check the process affinity and run some tests where you put only one job per socket versus packing the jobs together. $\endgroup$ – Jed Brown Jan 20 '12 at 20:20
2
$\begingroup$

Are you saying that when you run > 8 copies of the job on a single node, performance drops significantly?

This happens commonly for codes which require high memory bandwidth - that do a lot of reading and writing to memory for fairly modest amounts of computation. At some point you saturate the capability of the memory subsystem, and adding more tasks simply makes things worse. Doug Eadline has a good introductory-level discussion of this on his site ( http://www.clustermonkey.net//content/view/306/1/ ) -- he talks about things in terms of "effective number of cores", which I'm not sure is all that helpful a way to think about the underlying problem, but it gives you an idea of how many cores you could profitably use on a node.

You may be able to restructure your code to use less memory bandwith (using something like cachegrind might help if you're inadvertantly using more than you need) in which case you should be able to use a larger count of tasks per node without starving the other processes of memory bandwidth; or you might not, in which case you're stuck with using fewer cores per node.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks. Do you happen to know if type_a->type_b->type_c is more expensive than direct access to type_c? (c++) $\endgroup$ – Yotam Jan 21 '12 at 6:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.