10
$\begingroup$

I'm trying to optimize an MPI application with a highly asynchronous communication pattern. Each rank has a list of things to compute, and sends messages as necessary if the inputs or outputs reside on a different rank. In addition, each rank is threaded (currently with one communication thread and 5 workers).

I've instrumented the code with timers around the different performance critical portions of code, which gives me a list of (start,end,type) triples for each thread. Plotted in the obvious way, with time as the horizontal axis, rank and thread as the vertical, and color indicating what each thread is currently doing, I get an image like this for 16 ranks with 6 threads/rank:

Pentago rank and thread history

My question is: what are other ways of visualizing this data that might help pin down performance issues? Does anyone have a favorite type of plot they use when profiling asynchronous applications?

This data set is limited in that it doesn't know the dataflow structure, but I'd like to get as much insight out of it as possible before trying to collect something more complicated.

The uncompressed image is here in case anyone wants to look around (it failed to upload through the normal route). Unfortunately, Firefox doesn't accept it even though I believe it's valid, possibly because it's simply too large.

Improve this question
$\endgroup$
2
  • $\begingroup$ I had quite a bit of trouble getting my browser or almost any other program to load the large image. In the end, gimp did it, but you may want to re-consider the size or file format options. $\endgroup$
    – Pedro
    Commented Sep 9, 2012 at 8:28
  • $\begingroup$ Sorry about that. I think the image is valid, since Firefox gives me the same errors are running it through convert (ImageMagick). Possibly it exceeds some arbitrary size threshold. $\endgroup$
    – Geoffrey Irving
    Commented Sep 10, 2012 at 6:02

3 Answers 3

Reset to default
4
$\begingroup$

I spend a lot of time writing and debugging parallel code, both with shared and/or distributed memory, but without knowing your specific problem, I can only tell you what works best for me.

Knowing which routines take how much time is an important thing if you're looking at computational efficiency, but if you're worried about parallel efficiency, then you should be more worried about what your code is doing when it's not doing any computation. Kind of like worrying what the kids are doing when it's too quiet...

Since you're using a hybrid shared/distributed memory approach, I guess your code is, in the blanks, either waiting on an MPI call or on a mutex/condition variable. You can wrap these calls in timers too, and that will give you a better picture of what is slowing you down, e.g. if it's always the same conditional or always the same MPI_REDUCE that your threads get stuck on.

One piece of software I use quite often is the Intel Vtune Amplifier XE. It has a nice plotting feature/option that visualizes thread concurrency. The program will draw a plot very similar to yours, yet when a thread waits on a mutex or condition variable, it draws a diagonal line from the waiting thread, at the time it started waiting, to the thread that actually released the mutex or signalled the condition it was waiting for, at the time that it was released/signalled. This can be quite messy, but it makes bottlenecks appear immediately.

Finally, I also collect bulk statistics, e.g. for each mutex/signal/MPI-call, what were the average and maximum wait times? What is the histogram of the collected wait times? While the plot gives you a nice overview, it can get quite messy when it comes down to the fine details.

Finally, one question that should not be underestimated: How are you collecting your timings? Is your timer non-intrusive enough to not influence your code? I use the CPU instruction count whenever possible, i.e. RDTSC on x86 architectures. This usually just adds a single instruction to your code.

Improve this answer
$\endgroup$
4
  • $\begingroup$ The data already has blocks around all the waits; in the diagram they show as white for idle worker threads and yellow for waiting communication threads. Unfortunately, all waits in the communication thread occur in a single blanket MPI_Waitsome due to the asynchrony. Vtune doesn't apply in this case since the purely threaded performance is essentially perfect, but thanks for the pointer. The histogram suggestion is a good one too. $\endgroup$
    – Geoffrey Irving
    Commented Sep 10, 2012 at 6:08
  • $\begingroup$ As for timing overhead: I'm using gettimeofday, which is necessary at least around the idle sections since there I use pthread condition variables. Can CPU instruction count be made to work in such a situation? The overhead is already low enough, but lower would certainly be nicer. $\endgroup$
    – Geoffrey Irving
    Commented Sep 10, 2012 at 6:11
  • 1
    $\begingroup$ @GeoffreyIrving: Yes, you can use them, but they only make sense on CPUs that have the constant_tsc flag set (check /proc/cpuinfo) and if you use lock each thread to a specific core, i.e. each thread always reads the same register from the same core, e.g. using pthread_setaffinity_np. Note that the latter is Linux-specific and thus not portable. $\endgroup$
    – Pedro
    Commented Sep 10, 2012 at 11:16
  • $\begingroup$ @GeoffreyIrving: Even if you are waiting for some undisclosed event using MPI_Waitsome, you can still record what requests actually arrived and from where. This information may or may not be of use... $\endgroup$
    – Pedro
    Commented Sep 10, 2012 at 11:18
5
$\begingroup$

Sometimes you can get an alternative view on performance problems via a high-level resource analysis: Is there a relevant bottleneck such as memory bandwidth? Does every worker thread do the same amount of work? This data might be collected easily with likwid-perfctr from the LIKWID tool suite LIKWID Google code project. If the profile is such that many different hot spots exist, you may be required to tackle them one by one. There may also be different issues, depending on how many threads/processes are used.

Improve this answer
$\endgroup$
1
  • $\begingroup$ In the interests of perfect disclosure, Georg works on the LIKWID project and I solicited this response because I wanted to complement Pedro's great answer with another perspective (and a great, freely available tool). $\endgroup$
    – Aron Ahmadia
    Commented Sep 11, 2012 at 8:15
2
$\begingroup$

When I have a problem in a network of highly asynchronous processes governed by messages or events, I use a method which is not easy but is effective. It involved getting time-stamped logs of the processes, merging them into a common timeline, and tracking the progress of some messages as they trigger activities, triggering further messages. What I'm looking for is delay between the time a message is received and the time it is acted upon, and understanding the reason for the delay. When a problem is found, it is fixed, and the process is repeated. This way, you can get really satisfying performance.

It's important to see how this differs from approaches where you measure, measure, measure. The only thing measuring can conceivably tell you is where not to look. Real performance tuning requires looking carefully at the details, from a time perspective. What you're looking for is not where time is spent, but where it is spent unnecessarily.

Good luck.

Improve this answer
$\endgroup$
2
  • $\begingroup$ In other words, there is no useful visualization of the data I have. :) Jed Brown suggested Jumpshot (and associated utilities) as one way to collect and visualize the data you suggest, so I'll look into that. $\endgroup$
    – Geoffrey Irving
    Commented Sep 12, 2012 at 18:59
  • $\begingroup$ @Geof: Good luck with visualization. The only tool I would have found useful is something to collect and merge the event logs so I could follow the path of one or more requests as it made its way through the various threads, 'cause that's the only way I know of to detect unnecessary delays. That's what any performance problem will consists of - unnecessary delays. $\endgroup$
    – Mike Dunlavey
    Commented Sep 12, 2012 at 21:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.