4
$\begingroup$

I have a problem with parallelization and the brownian dynamics (molecular dynamics) code that I am using. We have our own home-grown framework at the university, and recently we've made the change to using OpenMP for the more intensive calculations. We are definitely CPU bound, so I don't necessarily have to worry about the total memory or disk speed of the computation, only the processor. There are multiple portions of the code that are parallelized, and they appear in the two basic forms.

  1. No shared variables, so no critical operations needed.
  2. An omp lock, so that we can access shared variables.

Also relevant is the architecture that these are running on. The cluster that I am using has 2 Intel Xeon hex-core processors, so 12 cores per node, and I only run on 1 node for a sim at a time. The problem is that when I move beyond using 6 cores on a node, I immediately see a performance hit, so I see worse performance using 12 cores than 6. Initially I thought that this was probably a synchronization issue between the two processors, but only 1 of my functions has the critical section, the other 3 don't have any shared variable, and also synch up their main results at the end. In this case, each thread has it's own array that it writes to, and then after they all complete the results are computed.

I did just move from using gcc to icc (intel compiler), which has definitely improved my times, but I was wondering if anybody had any thoughts on what else I could do to check out what is going on. Unfortunately, we don't have VTune installed on the cluster, so I can't profile using the Intel tools, but what does everybody here think? Does this sound like a caching issue to you? Or is it something more sinister.

I just don't quite understand that, given how little of the code has to synchronize, there is worse performance with more cores once I pass one processor.

Improve this question
$\endgroup$
4
  • $\begingroup$ A lot depends on how long a unit of work on a core actually runs, before you incur the overhead of finalizing that task and getting another one started. It's best if each one can sail along for a good long time before having to do any synchronization. Also, I've seen slowdown when the number of processes exceeds the number of cores, because then it has to do swapping. $\endgroup$
    – Mike Dunlavey
    Commented Feb 19, 2014 at 21:29
  • $\begingroup$ I'll take a look at how long the units of work run, but it should be fairly long, and when using gcc with openmp I set the wait policy to active, so that a thread doesn't necessarily yield the core. Most of the operations don't have any synchronization, and I'm using the nowait clause on my loops. I know that there are 12 real cores on the machines, so I could understand if using 10 cores was better, but again, the jump seems to be at 6, not 10 to 12. $\endgroup$
    – NuclearAlchemist
    Commented Feb 19, 2014 at 21:39
  • $\begingroup$ How often do you lock/unlock in your parallel loops? Does each thread lock/unlock different variables or the same variable? It would help if you could localize the most expensive parallel bit using a profiler and describe what it does, and how it does it, in pseudo-code. $\endgroup$
    – Pedro
    Commented Feb 19, 2014 at 22:52
  • $\begingroup$ I only lock/unlock when a certain random probability is found to be true, and then it's just the one omp_lock_t. There aren't even many instructions in that portion of the code - it is literally just decrementing a variable to keep track of how many things I've removed. As far as how many times this probability is hit, I don't know, but I can find out. $\endgroup$
    – NuclearAlchemist
    Commented Feb 19, 2014 at 23:14

1 Answer 1

Reset to default
4
$\begingroup$

It's possible that all the memory being used by your code is on one socket and that up to 6 cores all the tasks are running on that socket. When you get to 7+ sockets, then there are transfers between sockets to get at the memory. You may need to investigate memory affinity options for your threads. The default policy in Linux is first-touch (I think), so if you have one thread initializing things and then all the threads go to work, you may need to change this.

Improve this answer
$\endgroup$
10
  • $\begingroup$ I'll check this, since that would make sense. The memory affinity could definitely be a problem, although my code is small enough normally to fit into the L3 cache on the chips (minus other stuff). So I don't know, but I will check into this. $\endgroup$
    – NuclearAlchemist
    Commented Feb 19, 2014 at 23:17
  • $\begingroup$ Well, if thread 7 on socket 1 is accessing some memory that was originally touched by thread 0 on socket 0, then even if it fits in L3, you'll have to load it across the bus at the very least. $\endgroup$
    – Bill Barth
    Commented Feb 20, 2014 at 2:15
  • $\begingroup$ I was just looking at this. It seems like I could circumvent this by using OMP_PROC_BIND=true, which should bind the threads to the cores. As well, for the intel code, I can use KMP_AFFINITY="verbose,compact" if I was reading the documentation correctly. But I might still have the problem when they are attempting to access this same variable, as you pointed out. $\endgroup$
    – NuclearAlchemist
    Commented Feb 20, 2014 at 3:47
  • $\begingroup$ Thanks for accepting the answer, but did it work? $\endgroup$
    – Bill Barth
    Commented Feb 20, 2014 at 13:16
  • $\begingroup$ I'm still trying to figure that out. The supercomputing cluster lost power, so I need to restart all of my benchmarking tests here. I'm trying to profile this at home, but since it doesn't have the same architecture, it's pretty hard to tell if something is going wrong (I only have 1 proc). I am hopeful for the memory binding options in gnu/intel, but do you have a favorite profiler to use in this sort of environment? I also have to fight non-determinism, since the simulations can take differing amount of times depending on the initial random seed. $\endgroup$
    – NuclearAlchemist
    Commented Feb 20, 2014 at 15:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.