11
$\begingroup$

Concurrent to my research on Krylov Subspace Methods, I have the option of exploring mathematics behind HPC a step ahead or the theory of computation (hardware, OS, compilers etc.). Currently, I know both enough to just get by. For instance, I know how to derive the equations for CG and the basics of iterative methods but I am clueless about the details and more complicated stuff like Preconditioners and Convergence. Similarly, I know the basics of Finite Element Method (Weak Form, Non weak form, stuff like Codomain and Galerkin and stuff) but won't know the depths of it. On the computational front, I know how to code serially in all possible languages and can use OpenMP and MPI sufficiently well. I don't understand hardware and caching all that well.

My question is: What should one concentrate on: Math or Computation? Are they inseparable in HPC? Is it recommended for one to learn about one and not the other?

EDIT : I am currently majoring in Mechanical Engineering (which I regret) and have tons of courses in engineering and computation (fluids, heat transfer and so on). I will be joining graduate school for HPC this year and I want to strengthen some aspect (Math/Comp/Hybrid) before I begin my grad studies. I like math and comp equally (so the "Do what you enjoy more" is redundant).

Improve this question
$\endgroup$
0

5 Answers 5

Reset to default
10
$\begingroup$

I like aeismail's answer, but I'm going to provide an alternative perspective.

In optimization, it's impossible to really learn the field without understanding real analysis. Even before you tackle numerical issues, you need to understand notions of convergence of sequences, because you are going to prove in classes that algorithms converge. You're going to have to understand concepts like continuity and differentiability on more than a superficial level. Consequently, real analysis is a prerequisite for courses in nonlinear programming.

My thesis relates to methods for solving ordinary differential equations. Convergence issues, specifically things like "if I reduce my local error tolerance, then my calculated numerical solution approaches the true solution of the equations I'm solving" are again issues that require real analysis. To develop the theory for convergence issues required me (against my advisers' wishes) to take two semesters of real analysis. (It paid off with a couple manuscripts.)

However, I know there are people out there who survive quite nicely in numerical methods and HPC without taking pure mathematics classes. It really depends on the niche that you want to occupy.

If you want to develop new methods, then theory classes are helpful. Theory classes are also helpful for general mathematical literacy; reading math papers becomes much, much easier.

If you want to apply specific numerical methods to problems, numerical methods classes are more helpful. I believe this perspective is where aeismail is coming from, and it is a situation more common for engineers. (Disclaimer: We know each other, and graduated from the same department.)

As for HPC, the impression I've gotten is that experience is the best teacher. I took a parallel programming course, and it was slightly useful, but the main message of the class was to try things and see if they worked. If it's important for your thesis research, you'll get experience in HPC. If it's not, you won't, and it probably won't matter until you want to switch gears and tackle HPC problems. My thesis hasn't been especially HPC-heavy, at least in terms of what I program, so I haven't needed to pick up that set of skills.

To wrap up, you should probably concentrate on getting background in issues that relate to your thesis problem, keep in mind what you think you want to do in the future, and decide what broad, general background you need to communicate with other researchers in the community you'd like to join. Your PhD is going to be one of the last opportunities for you to take classes, and if you think you do want to learn math theory (or any subject, really), learning it on your own is considerably harder without establishing some sort of basic proficiency first.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Interesting perspective—and the example is a useful counterclaim to my point. (I should point out that I'm about to give a few lectures on optimization where I am explicitly leaving out proofs of convergence, because the focus is on the numerical methods, and there really isn't enough time to "prove" things if I have to introduce real analysis as part of the bargain.) $\endgroup$
    – aeismail
    Commented Jan 8, 2012 at 21:29
  • 2
    $\begingroup$ I recommend taking enough mathematics courses to be able to understand theorems and (when necessary, with effort) proofs appearing in journals like SISC, J. Scientific Computing, CMAME, etc. This probably means a course in real analysis, a course in abstract PDE theory, a course in general numerical analysis, and a course in discretizations for partial differential equations. In my personal experience, self study, digging around in open source libraries to understand why choices were made, and most importantly, becoming a developer of such a library (PETSc) were invaluable for learning HPC. $\endgroup$
    – Jed Brown
    Commented Jan 9, 2012 at 16:25
  • $\begingroup$ Jed: Sadly, this isn't possible in the context of many graduate students. I know I wouldn't have been able to take all of those courses, plus all of the physical sciences courses I needed for my direct field of research. So how does one balance that—particularly in the context of having an advisor who may not want a student enrolled in (or sitting in on) lots of courses? $\endgroup$
    – aeismail
    Commented Jan 9, 2012 at 23:05
  • $\begingroup$ @aeismail: I viewed it as an investment in career, and there are still gaps I need to fill. If it's important, you find the time to do it. (Like I said, I did it against my advisers' wishes, and ended up with an approach that begins to solve the type of problems they said they wanted to solve for the past 10 years.) That said, it's definitely hard to find the time, and it's hard to find advisers who are supportive when there's so much pressure to publish. It's also hard if advisers aren't in computational science (or they have different ideas about what it is than you do). $\endgroup$
    – Geoff Oxberry
    Commented Jan 9, 2012 at 23:25
12
$\begingroup$

HPC is a blend of mathematics, computation, computer science, and applications. You need to be able to understand all of them to be truly successful in the long run. However, you do not necessarily need to achieve the same level of proficiency in all of them.

In the computation versus mathematics argument, for an engineer, I'd argue the numerical implementation issues are more important at first. If you wait until you've learned the mathematical theory and then begin implementation, you may spend a long time working on things that, while no doubt useful, may not directly impact your thesis research.

So, I'd lean towards understanding the computational aspects at first, and then going back and filling out the holes in the mathematical theory. Hardware issues can also be learned--but a lot of how that affects software will also be platform-dependent, so again, it may not be the first item on your agenda.

Others may of course disagree with me; as you stated, this is more of an opinion piece than a factual question.

Improve this answer
$\endgroup$
9
$\begingroup$

Take as many courses as you can in both. I did, and I don't regret it.

Assuming you're interested in a research career, you can be successful with any mix of the two. Find collaborators whose knowledge complements yours. I know a significant amount of mathematics as it relates to accuracy and stability of numerical methods, but much less about HPC. I have collaborators who know HPC very well, so working together we can get innovative numerical methods running on big machines. I do the math and they do the computation, for the most part.

That said, I think that math

  • is more fundamental
  • is more challenging to learn
  • remains relevant for a longer time period

whereas HPC topics

  • change more rapidly
  • can be picked up more easily on your own
  • are less generally useful and more problem/application/machine specific

This is an overgeneralization and will surely attract disagreeing comments. But I think there is truth to it.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you very much for all your answers. Considering everything, among the 3 pillars of CSE(Numerical Math, HPC and Applications to Science/Engineering). I am interested in all of them but shied away from Math because I couldn't follow the proofs and papers well. By focussing on Real Analysis, Linear Algebra and Numerical Methods now, I think I'll prepare myself for everything. My advisor said the level of pure calculus one understands is directly proportional to the level of appreciation for any applied field. As I am (re-)reading Calculus after years of Engg, I am convinced of the aphorism. $\endgroup$
    – Inquest
    Commented Jan 9, 2012 at 15:19
6
$\begingroup$

I agree with both aeismail and Oxberry. I decided to write an answer because you seem to be facing the same questions I was trying to find an answer to last year. I also majored in mechanical engineering (and hated it, specially solid mechanics), I spent a lot of time working with numerical methods in CFD or optimization. Now I am doing my masters in Applied Mathematics and Computational Sciences. From my point of view you first need to decide on what you want to do in the future. If you want to go into modeling or development of numerical methods then you should definitely go towards math. I spent two years working with Finite Volume and Finite Element methods without knowing the deep basis and now that I am taking classes in applied math it is all making a lot more sense to me. I realize how the methods work exactly and I am no longer walking blindly just experimenting with everything. It saves a lot of time and effort. But if you decide you want to go into developing software and related topics then you might want to focus on the HPC part. In my experience, there are a lot of packages out there that are optimized and ready to use for a lot of numerical applications. So it won't be the best idea for me to spend a lot of time developing my own software so I decided to work more on the math part.

Improve this answer
$\endgroup$
1
$\begingroup$

I don't believe in a theory/application dichotomy, but it's also important to approach fields in a way that is not completely out of context. Understanding theory I think provides you with a general intuition on the problem that is very valuable as it keeps you from having to focusing on concrete after concrete (i.e. one particular implementation versus another), and lets you look at the big picture. This understanding however doesn't arise out of a vacuum, and you can't BEGIN at this level.. that's not how the brain works. You can't arrive at the concept of a forest without having ever seen a tree!

That is not to say that theory takes a subservient role in this question, either. It is to say that theoretical understanding provides one with significant mental economy when considering a class of problems, but it can't exist without the concretes that drive it (at least in the computational theories).

So to answer your question: If all you are interested in is the implementation, that is the result, but not in improving/altering this implementation, theory won't be that important. If however you wish to produce your own, then you are at a competitive disadvantage with those who understand the theory better. Unless of course you produce your own over the years that happens to be better :)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.