Apologies if this is a bit of a soft, unclear, or opinion-based question. I'm a relatively new PhD student in a (computational) quantum chemistry group. My group develops and maintains a few software packages (mostly for techniques first developed in-group), and this is something in which I am interested, i.e. high-performance numerical algorithms and implementing them. I did a minor in computer science and an honours in mathematics but I never really explored computer hardware/architecture (mostly because classes did not fit in my schedule), so I've only really studied the "high-level" area of computer science (optimisation, algorithm analysis, machine learning, etc.). However, in my work so far I am already coming across a lot of what appears to be hardware-related issues (of course, lots of discussions about threading and concurrency; I am also trying to learn about GPU acceleration). So I'm wondering, how important is understanding computer architecture (and that "stream" of classwork: operating systems, hardware, distributed systems, etc.) for a practising scientific software developer? (especially in the field of quantum chemistry)
I understand knowing these things won't hurt and that we certainly rely on them, but what is a good working knowledge for a professional, so that I know how much time to spend on learning them? I would guess that in principle I could get by without knowing much at all and just rely heavily on libraries, but I also guess that understanding how they work, identifying bottlenecks, etc. would be beneficial, not to mention open more avenues. I'd also appreciate any suggestions for resources (e.g. textbooks, MOOCs) for developing a working knowledge in these areas.
Edit: For clarity, I have not taken any courses on hardware, but I have learned vague ideas about memory as needed in work and in algorithm classes. I've also read the first four chapters of The Elements of Computing Systems by Nisan and Schocken (and did the exercises) in my spare time a few months ago. My long-term career aspiration is probably to stay in academia, but I am not totally sure. I am also interested in working in industry (probably still R&D) and HPC centres, or at least want to be conversant with them such that I would be able to collaborate from academia should I stay in it.