5
$\begingroup$

Suppose I want to set up an experiment to measure the performance of some numerical code on a desktop machine running Linux/Windows/MacOS. What kind of environment should I arrange in order to get correct timings? More precisely,

  • what applications/services should be closed?
  • what priority the considered process should have?
  • how many runs should I typically perform to average the timing?
  • are there any specific issues with parallel CPU/GPU computations?
  • anything else?..

Of course, all these issues are platform specific. The interesting question for me is also how Linux and Windows compare at that point.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
12
$\begingroup$

I think it depends on your purpose. If you are trying to assess the overall performance of the code or environment, then I'd encourage you to run it however you think most people will run it on a desktop environment: leave things open but make sure nothing is crunching in the background or hogging all the memory. The biggest culprits, in my experience, are backup tools during archive cycles and browsers with lots of and/or runaway tabs. If you're afraid that the performance difference is getting lost in the noise, then it probably isn't significant from a desktop user point of view.

If you are trying to evaluate the performance of a specific, potentially small code or environment change, then there is a trade-off: you can be more careful with shutting everything else down to reduce noise or you can try to isolate or emphasize that change to boost the `signal' in your experiment. I tend to prefer the latter, but your mileage may vary. This is related to profiling, for example here and here.

Someone else can give a better answer for getting the most precise timings, but for desktop experiments, I think it is overkill.

Things to look out for:

  • Hyper-threading can lead to confusing results. Though, if users won't bother turning it off, it could be relevant for overall time. (disable in BIOS)
  • Turbo-boost poses similar issues to hyper-threading. (disable in BIOS)
  • Threaded BLAS libraries can make 'single core' runs parallel (disable with OMP_NUM_THREADS=1 in environment)
Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ Another big danger is variable-speed processors such as Intel Burst. $\endgroup$
    – Aron Ahmadia
    Commented Jan 7, 2014 at 17:24
  • $\begingroup$ @AronAhmadia For looking at specific parts of the code: absolutely, but if users aren't going to turn it off in the bios, I think its relevant for overall performance. Very similar situation to hyperthreading. I'll add it to the list. $\endgroup$
    – Max Hutchinson
    Commented Jan 7, 2014 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.