I like aeismail's answer, but I'm going to provide an alternative perspective.
In optimization, it's impossible to really learn the field without understanding real analysis. Even before you tackle numerical issues, you need to understand notions of convergence of sequences, because you are going to prove in classes that algorithms converge. You're going to have to understand concepts like continuity and differentiability on more than a superficial level. Consequently, real analysis is a prerequisite for courses in nonlinear programming.
My thesis relates to methods for solving ordinary differential equations. Convergence issues, specifically things like "if I reduce my local error tolerance, then my calculated numerical solution approaches the true solution of the equations I'm solving" are again issues that require real analysis. To develop the theory for convergence issues required me (against my advisers' wishes) to take two semesters of real analysis. (It paid off with a couple manuscripts.)
However, I know there are people out there who survive quite nicely in numerical methods and HPC without taking pure mathematics classes. It really depends on the niche that you want to occupy.
If you want to develop new methods, then theory classes are helpful. Theory classes are also helpful for general mathematical literacy; reading math papers becomes much, much easier.
If you want to apply specific numerical methods to problems, numerical methods classes are more helpful. I believe this perspective is where aeismail is coming from, and it is a situation more common for engineers. (Disclaimer: We know each other, and graduated from the same department.)
As for HPC, the impression I've gotten is that experience is the best teacher. I took a parallel programming course, and it was slightly useful, but the main message of the class was to try things and see if they worked. If it's important for your thesis research, you'll get experience in HPC. If it's not, you won't, and it probably won't matter until you want to switch gears and tackle HPC problems. My thesis hasn't been especially HPC-heavy, at least in terms of what I program, so I haven't needed to pick up that set of skills.
To wrap up, you should probably concentrate on getting background in issues that relate to your thesis problem, keep in mind what you think you want to do in the future, and decide what broad, general background you need to communicate with other researchers in the community you'd like to join. Your PhD is going to be one of the last opportunities for you to take classes, and if you think you do want to learn math theory (or any subject, really), learning it on your own is considerably harder without establishing some sort of basic proficiency first.