HPC reading material

Question

I am an undergraduate, and enthusiastic about HPC. I am currently familiar with the tools OpenMP, MPI, CUDA, OpenCL, thrust libraries etc. But I want to know the core functioning of these tools, I want to know how exactly these environments work to get the most of the hardware. It would be great if someone would suggest me some reading material or even some MOOC videos.

After reading the material it would be great if I could get a very good idea how the library is written. And maybe someday even I could write one if needed.

Thanks in advance.

Answer 1

The Intel OpenMP Run Time library has been open sourced, so you might have look at a few functions in there. There are many open source MPI libraries (MPICH2, OpenMPI, MVAPICH2, etc.), and the MPI interface is standardized, so reading the standards document can be enlightening. OpenMP is also standardized, so you might look to that as well. CUDA is basically a proprietary NVIDIA language, so don't expect to get too much information about its inner workings. Thrust is also open, so you can poke around at it on GitHub if you like, though you might spend some time looking at the C++ STL and Boost before you do that.

Answer 2

This question has appeared a long time, anyway I cite the book:

Rauber, Thomas; Rünger, Gudula, Parallel programming for multicore and cluster systems, Berlin: Springer (ISBN 978-3-642-37800-3/hbk; 978-3-642-37801-0/ebook). xiii, 516 p. (2013). ZBL1295.68012.

I cite two extracts from the book description:

Rauber and Rünger take up these recent developments in processor architecture by giving detailed descriptions of parallel programming techniques that are necessary for developing efficient programs for multicore processors as well as for parallel cluster systems and supercomputers.

The main goal of the book is to present parallel programming techniques that can be used in many situations for a broad range of application areas and which enable the reader to develop correct and efficient parallel programs.

It is not related to a specific library but I found it a good book for the parallel part of the HPC world.