How should I study creating and programming HPC systems?

Question

I'm in a field that doesn't necessarily do a great deal of HPC work, and when it does encounter it, it's often the result of researchers from other fields exploring new applications to their methods and the like. Primarily what this means is that it never really gets introduced in the course of study, or get brought up much at workshops, seminars and the like - it's possible to go one's entire career without needing it.

At the same time however, much of the work I do could benefit from making better use of the HPC resources available to me - mostly in the form of nicely parallel Monte Carlo simulations.

My problem is finding resources for learning how to go about using clusters, MPI, etc. And separating the good from the bad, given I don't know much.

Any suggestions for books either on programming on these types of systems, or on setting up and running very modest HPC setups of one's own?

Answer 1

Check out HPC University. In particular, the resources section, which includes things like

• an Introduction to OpenMP
• Debugging Serial and Parallel Codes
• Introduction to the Open Science Grid

and much more.

There are many higher education programs that include courses in HPC. As an example, my own program includes courses in

• High performance computing (2 semesters)
• Parallel programming paradigms
• Scientific software engineering
• GPU and GPGPU Programming

as well as a variety of courses in scientific computing and numerical analysis.

Answer 2

My personal experience is that HPC requires such a broad set of skills that is very difficult to do it in just one book, recipe, university course. Consider that HPC is not about programming big machines or building a cluster but rather tackling big science problems at the bleeding edge of computing.

I'm not sure everyone would agree but I think that a list of areas involved in HPC could be:

• Computational models, simulations, numerical analysis, and algorithms
• Scientific modelling in fields like applied mathematics, physics, engineering
• Computer architectures and programming
• Scientific visualization

Each of this areas are immensely large and there is so much knowledge that is difficult to get hold of everything before attempting to do anything!

In my case, I started from one corner of engineering and slowly developed experience and a collaboration network. At the very beginning I just had one engineering / applied maths problem that I wanted to solve, from there I moved to programming, to getting more performance via more powerful computers, to numerical analysis, to advanced algorithms, to parallel computers, to parallel algorithms, to even larger computers, to a more complete scientific model, and eventually to a new problem and was iterating again. If I look back, trying to understand and solve my first problem was what crafted my study plan. Having a supervisor that can keep you on the right track also helps a lot, otherwise be careful of not loosing sight of your end goal (in my case, this was the science problem I had at hand).

If I have to name one thing that really makes a difference while going down the HPC path was to meet people from other areas, exchange ideas, and take advantage of the distributed knowledge of the community. Also, it is important not to be too afraid of being openly naive and doing silly things. The first steps are always difficult and the most important ones!

Answer 3

This is my plan for teaching myself some HPC foundations this coming spring. I've been a developer for the past decade and half, and work exclusively with Microsoft technologies. I'm also a student, so I can get some software free through MSDN Academic Alliance (your profile indicates you're a grad student, so you should check to see if your school participates), and making room for stuff at home is the limiting factor. My office/den/ManCave already has a number of computers in it so I can already test some stuff. Windows Server 2008 HPC is listed in some available packages, but it seemed to disappear off my school's downloadable software during the past semester ("cluster compute pack" is an add-on for server 2003, while server 2008 hpc comes with a stripped down version of server 2008, with ccp, you'll have to disable services you don't need).

My planned topology is the "SOA Enabled Cluster" which is the 4th image on this page (click the tiny squares under the main picture). I already have Active Directory and some other servers already set up, so I'm thinking about getting 4 cheap Dell small form factor computers off eBay (refurb Dell 655 SFF appear to be running in the $100-200 ballpark). Since I'm only trying to learn some stuff, I believe I only need a "head node" a "broker node" and 2 "compute nodes" and a few spare ethernet cards to see what the private and application busses (from that image) do. The MSDNAA license permits up to 16 compute nodes which is far more than I'm looking for.

I've been reading Beowulf Cluster Computing with Windows, and I think I've got an idea what I'm up against for some ideas at the office (there is a linux version of the book, but I have 0 linux skills/platforms at home - do what is right for yourself). There are a lot of big clusters available at the office, but they're already saturated, and educational "I want/need to learn X" time slots are very hard to get. One is already expected to know the stuff before one can submit jobs.