I am a PhD student and my lab has developed a code to run simulations. It relies on external libraries and code my lab has written. I've run a strong scaling study which shows poor performance, see the plot attached.Strong scaling.

We know that the code should at least be close to the Good efficiency (the method should scale well, the parallel part of our library relies on another library which scales well). In order to find the cause of the problem, I ran several profiling runs with Intel vtune (hotspot analysis), but the results do not show anything unexpected. I also run a strong scaling study (with the help of vtune) on a function level, hoping that it would show me which function performs worse than the others and it would at least give some pointer as to where to start. However, the top 10 most time intensive functions all perform the same, poorly.

I was wondering what other strategies I can use to find the bottleneck?

Edit: As per @Tyberius recommendation, I performed a strong scaling for a lower number of cores (1,2,4 and 8 cores). Here, our code seems decent (PHiLiP). Does this mean that I should be looking at the MPI calls?

Edit2: We use dealii (9.3 pre) for the parallel part of our code. This is a well-known finite element solver. As for the size of the problem, I performed a weak scaling which gave us a bad result as well. I suppose if the problem for strong scaling would be too small, the weak scaling would have given us a better result (?). It takes almost 4 hours to run this code on 1 core (we do a good amount of time stepping). As for the 256^3 case, we can't run that. We would run out of the 1 week cluster job time and we have some modifications to make our code run beyond that, it would away a good amount of computing time and we are still at the beginning of the year.

enter image description here

  • $\begingroup$ I did not look at memory bandwidth bottlenecks. I ran a different MPI profiler, which showed the communication and the various MPI functions, but I could not get more out of it. No, there is no GPU component, yet. $\endgroup$
    – nyaki
    Commented Jun 3 at 17:45
  • 1
    $\begingroup$ "the top 10 most time intensive functions all perform the same, poorly." If all your most time intensive functions scale poorly, it sounds like you have multiple bottlenecks. Assuming everything these functions are run sequentially and that parallelism is applied with in the functions (rather than multiple high level tasks being done in parallel) than improving the scaling for anyone of these time intensive functions should improve your performance. $\endgroup$
    – Tyberius
    Commented Jun 3 at 23:18
  • 1
    $\begingroup$ @Tyberius The graph starts at ten processors and measures speed-up against that base case. $\endgroup$ Commented Jun 4 at 2:51
  • 1
    $\begingroup$ You mention that the library you rely on scales well. Is that based on their benchmarks or your own? Can you construct a trivial example of a library call that should scale well and check to confirm that is the case for you? Is the example you are using to test the scaling big enough to really use all available cores 10s or 100s of cores? If not, eventually the overhead of managing parallel execution is going to outweigh the marginal performance benefit of adding an additional core. $\endgroup$
    – Tyberius
    Commented Jun 7 at 18:16
  • 1
    $\begingroup$ The label in your first plot days 16^3 elements. You did not mention what your code computes, but for some codes such a small case might be simply too small to show good scaling. How does your plot look like with lets say 256^3 elements? $\endgroup$
    – dweber
    Commented Jun 8 at 18:58

2 Answers 2



  • You could try two write timestamps at the beginning and the ends of your mpi communication instructions on each node. If the times to communicate the necessary data is linear, then your network is fast enough. It might just be saturation of some networking component. Sometimes we focus to much on CPU and not on the other aspects of computing.The bandwidth (i.e. expected mpi bandwith) between cores on the same silicon, the same mainboard and each server rack will differ. You (sadly) can not expect to have constant time communication when scaling further up.

  • Look up the memory sizes of your cluster, i.e. the L1/L2/L3 and RAM of each node and compare them to the expected memory footprint of your code. Sometimes these 'bumps' in scaling can be traced back to being able (or not able) to fit the node-assigned data onto these memory elements.

  • It helped me in the past to think about the parts of the code which can be parallelized, and those parts which have to be executed synchronously. There usually is a certain percentage of instructions which can not be distributed. When you identify those you may use ahmdal's law to refine your expectation of your scaling.

  • $\begingroup$ Thank you for your insight. I am not sure I understand what should I do with the timesteps. Before and after the main MPI calls, I write the timesteps in a file/stdout. How can I deduct information regarding whether it is linear or not? I will do the second point and see how it compares. $\endgroup$
    – nyaki
    Commented Jun 10 at 12:01
  • $\begingroup$ You seemed to focus on the benchmarking of the cpu time consumed by the different parts of your code. If you measure the communication times via some output ("timestamps") you may check if your networking takes as long as expected. Thats what i meant. $\endgroup$
    – MPIchael
    Commented Jun 11 at 2:15

I want to add to what @MPIchael said, particularly on their third point about Ahmdal's Law.

On one of my prior projects doing spacetime Galerkin stuff, we were doing scaling studies for our distributed software (written in C++) and found it was not achieving nearly the performance we expected. After a bit of work, we realized that some older serial parts of the software in the finite element side of the codebase had been written very poorly by some former PhD students. To our surprise, the software was packed with tons of inefficiencies that we never expected. Some of the biggest offenders were:

  1. Using std::map containers all over the place, most situations that could have been reduced to just using arrays.
  2. The former students used new allocations and delete[] calls all over the codebase in places it made no sense, even for things like just creating small 4x4 matrices.

The second offender was by far the worst and after some work by one of my colleagues, most of these allocations were removed and replaced with reusing previously allocated memory and using the stack (say for the 4x4 matrices), when it made sense. After these changes, the scaling performance grew enormously!

So it could make sense to try and profile the serial parts of your codebase and see if there are any places that could be sped up by improving data structure choices, reusing memory, and/or other non-trivial algorithmic choices. There could be some low hanging fruit here!

  • $\begingroup$ Inefficiencies in the serial part would not explain those plots, though; code that uses the heap is still expected to scale almost-linearly. $\endgroup$ Commented Jun 9 at 12:19
  • $\begingroup$ We have some inefficiencies in the serial part, but the most basic operations are performed by the library and we use their data structure. However, I will inspect these again, just to be sure. $\endgroup$
    – nyaki
    Commented Jun 10 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.