I have a problem with parallelization and the brownian dynamics (molecular dynamics) code that I am using. We have our own home-grown framework at the university, and recently we've made the change to using OpenMP for the more intensive calculations. We are definitely CPU bound, so I don't necessarily have to worry about the total memory or disk speed of the computation, only the processor. There are multiple portions of the code that are parallelized, and they appear in the two basic forms.
- No shared variables, so no critical operations needed.
- An omp lock, so that we can access shared variables.
Also relevant is the architecture that these are running on. The cluster that I am using has 2 Intel Xeon hex-core processors, so 12 cores per node, and I only run on 1 node for a sim at a time. The problem is that when I move beyond using 6 cores on a node, I immediately see a performance hit, so I see worse performance using 12 cores than 6. Initially I thought that this was probably a synchronization issue between the two processors, but only 1 of my functions has the critical section, the other 3 don't have any shared variable, and also synch up their main results at the end. In this case, each thread has it's own array that it writes to, and then after they all complete the results are computed.
I did just move from using gcc to icc (intel compiler), which has definitely improved my times, but I was wondering if anybody had any thoughts on what else I could do to check out what is going on. Unfortunately, we don't have VTune installed on the cluster, so I can't profile using the Intel tools, but what does everybody here think? Does this sound like a caching issue to you? Or is it something more sinister.
I just don't quite understand that, given how little of the code has to synchronize, there is worse performance with more cores once I pass one processor.