4
$\begingroup$

I know this question is somewhat familiar to my previous question found here, however I am now coming at it from a slightly different and more general angle.

Let's say I have a code whose runtime is in the days to weeks range. For one reason or another, the code can not be changed to improve performance. My goal is to purchase a computer to run this specific code as fast as possible. It is currently parallel is some regards, but must be shared memory.

Currently I have two computers on which to run it. They both run the same Linux distribution and both run on Intel-Xeon processors, however the CPU models are different as is the memory speed, and other hardware values (the computers have about 5 years of age difference between them). On both computers I can easily change the number of cores utilized for running the code, and the newer of the two supports hyperthreading.

My question is, is there a way to profile runs of the code on one or both computers that would allow me to find where the hardware bottlenecks are, and thereby tune a new computer purchase for optimal performance with this code.

$\endgroup$
6
$\begingroup$

I assume you're talking about server-class parts here. Client parts may be a bit different, but most of the logic holds.

  1. Find a test case that runs with one thread in ~10 minutes that's representative of your longer problems in that it exercises the important parts of your code in similar ratios. You may have to scale down the problem size in order to scale down the initialization time, etc.

  2. Since you only have Intel Xeons, try turning on and off hyperthreading and comparing the run times of a version of the code that uses the maximum thread count in both cases. If your code is modestly faster with hyperthreading turned on, then you should borrow time on an AMD Bulldozer-based machine to test to see if it's shared FPUs are advantageous for your code. It's not exactly the same, but if two hyperthreads can successfully share the Xeon FPU, then you might have a chance at some decent performance out of the shared FPU on an AMD machine.

  3. Use Linux or the BIOS's ability to set a sequence of fixed, but reduced CPU clock rates for runs of your code (e.g. 2.7GHz, 2.6GHz, ...). If the performance scales strongly with clock rate, without plateauing at the high end, buy the fastest clock rate CPUs you can afford. If the performance plateaus after some clock speed, then consider not buying the top bin CPU, but backing off and buying more or faster memory. If you at all can, benchmark your code on the actual hardware you want to buy. This may be hard if you only want to buy one node, but ask around, someone you know may have one you can borrow.

  4. Consider your computational plan. If the cases you need to run are finite and enumerated, calculate the cost of renting some EC2 instances from Amazon instead of buying a machine. If you're only planning on buying one node, you may find that even keeping it fully loaded, your planned runs will take longer than you can afford to wait, and buying 10x as many instances from Amazon and running them there may be more effective. Plus, it's actually hard to keep a single workstation or server node busy all the time. It requires diligence. Amazon nodes you can rent when you need and discard when you don't. You don't have to pay for time you're not using.

  5. Strongly consider applying for supercomputer time with an organization like XSEDE in the US or PRACE in the EU (or your country's local equivalent). We (I work for TACC which participates in XSEDE) literally give away the time, and ~100k core-hours can be had for free in a matter of days or weeks through a start-up allocation. I presume the complementary organizations have similar policies.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you for the excellent answer. Since I am at Ohio-State, my research groups gets a small amount of free time on the OSC clusters, however our issue there is that it is often useful to view both logfiles and the total output files (in the GB range) during a run. Not having quick, local access to these files is the main reason we are looking at our own single node, since our budget would buy one slightly better than the nodes at OSC. $\endgroup$ – Godric Seer Apr 17 '13 at 2:19
  • $\begingroup$ I presume you mean the log files of your code not the system logs. Given that, why don't you have access to a filesystem that lets you see such things? I've run on Glenn, and it seems configured to allow them. $\endgroup$ – Bill Barth Apr 17 '13 at 2:24
  • $\begingroup$ I am more worried about pulling down 3-4 GB data files every couple hours to check the progress. I agree that the code logfiles would be fairly trivial. Even so, it is still a very real possibility once we get to the stage with the code to be running many similar cases that don't need this level of baby-sitting (mesh studies and the like). $\endgroup$ – Godric Seer Apr 17 '13 at 2:26
  • $\begingroup$ I presume OSU has at least 1Gb/s links to OSC and that your lab is on such a link. At that rate, you ought to be able to pull down the data pretty quickly, and they won't mind. Also, given that Oakley has GPU nodes, perhaps you can check the progress using a visualization node and VNC without having to move the data anywhere. $\endgroup$ – Bill Barth Apr 17 '13 at 2:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.