How important is learning hardware/architecture for scientific computing?

Question

Apologies if this is a bit of a soft, unclear, or opinion-based question. I'm a relatively new PhD student in a (computational) quantum chemistry group. My group develops and maintains a few software packages (mostly for techniques first developed in-group), and this is something in which I am interested, i.e. high-performance numerical algorithms and implementing them. I did a minor in computer science and an honours in mathematics but I never really explored computer hardware/architecture (mostly because classes did not fit in my schedule), so I've only really studied the "high-level" area of computer science (optimisation, algorithm analysis, machine learning, etc.). However, in my work so far I am already coming across a lot of what appears to be hardware-related issues (of course, lots of discussions about threading and concurrency; I am also trying to learn about GPU acceleration). So I'm wondering, how important is understanding computer architecture (and that "stream" of classwork: operating systems, hardware, distributed systems, etc.) for a practising scientific software developer? (especially in the field of quantum chemistry)

I understand knowing these things won't hurt and that we certainly rely on them, but what is a good working knowledge for a professional, so that I know how much time to spend on learning them? I would guess that in principle I could get by without knowing much at all and just rely heavily on libraries, but I also guess that understanding how they work, identifying bottlenecks, etc. would be beneficial, not to mention open more avenues. I'd also appreciate any suggestions for resources (e.g. textbooks, MOOCs) for developing a working knowledge in these areas.

Edit: For clarity, I have not taken any courses on hardware, but I have learned vague ideas about memory as needed in work and in algorithm classes. I've also read the first four chapters of The Elements of Computing Systems by Nisan and Schocken (and did the exercises) in my spare time a few months ago. My long-term career aspiration is probably to stay in academia, but I am not totally sure. I am also interested in working in industry (probably still R&D) and HPC centres, or at least want to be conversant with them such that I would be able to collaborate from academia should I stay in it.

Answer 1

I haven't worked in quantum chemistry specifically, but I've worked in other areas where high performance is a correctness requirement (along with scientific accuracy), so I think we're on the same page here.

Broad but shallow knowledge of all of the above is absolutely necessary for the team as a whole. Deep knowledge can be acquired as needed, or hired as needed.

Someone needs to know a little about operating system scheduling. Someone needs to know a little about CPU pipelines. Someone needs to know a little about distributed filesystems. Someone needs to know a little about compilers. They do not all need to be the same person.

As for advice on how to learn, I honestly don't know what's the best option today. If you're really starting from scratch, and you have access to a university library, I'd go with a classic, such as Hennessy and Patterson's Computer Architecture: A Quantitative Approach.

Answer 2

I want to support the response from @Pseudonym, who makes the point that not everyone in the team needs to contribute to every aspect of the project. Something related to consider is that you are presently at the beginning of your career, and will be making whatever contribution you are capable of. But perhaps you will still be working in the same general field in five years time , or ten years, or fifty. By then, perhaps some other aspect will have come to seem more interesting, or more important, so perhaps you will be concentrating on that, but your present work will provide experience and perspective that may allow you to lead the team.

A prospective employer is probably not going to regard your PhD as chiefly a guarantee of expertise in some narrow topic, but more as evidence that you can work independently. Many people eventually achieve fame in areas quite remote from their thesis work. You still have lots of time to carve out your distinctive career. You do not need to decide now what will be most useful in the future.

Answer 3

Ideally in your education you learn various subjects out of curiosity and in order to discover, if you're lucky, those you really like. And then you study those primarily because you like them. One or two of them may continue to intrigue and stimulate, and you end up knowing them inadvertently to a very great detail.

Studying what you think is needed for your profession isn't sustainable, and your brain will start resisting going deeper into something it doesn't enjoy. All these subjects tend to become rather complex, challenging to comprehend for a brain uninterested, unfueled by passion.

So, above all useful education, it's important to find one or two subjects you really like and be faithful to them, be it quantum chemistry theory, lab experiments, computational algorithms and simulations, or modern high-performance programming frameworks.

Which book or video lecture is there that you can't resist reading, watching, or programming at night for your own irrational, guilty fun when it's long past your bedtime?

Answer 4

On a team it is very common for technical help to be brought on as collaborators in developing low-level implementation. If you are a solo developer, or leading a team, these things become more important. In such a case you'll need a jack-of-all-trades familiarity with many steps of the computing hierarchy.

Crucially though, that shallow familiarity will mainly be useful in acting as a platform from which you can reach out to experts and specialists for help. Either way- solo or team- you will benefit from knowing the concepts and technical language enough to understand what to ask, how to ask effectively and how to incorporate the input of technical experts. The easier it is for you to know what to ask and who to ask the less time you will spend stuck. In my experience efficiently getting yourself unstuck is an immensely valuable and central skill for research.

Now for references- If you're at the point where you haven't firmly connected hardware to software conceptually I'd recommend But How Do It Know? by J. Clark Scott. It's a quick read that builds up a CPU from scratch with a fantastic level of readability and accessibility. That won't help you interface with Open-MPI on a computing cluster, but it will dispel the fog of mystery between metal and code.

From there if you want a more technical walk up the chain I'd recommend Computer Systems: A Programmer's Perspective by Bryant and O'Halleron. Working through that book would leave you with the language and concepts that will serve you well with whatever else may come up.