Consumer hardware for scientific computing?

Question

I'm interested in problems around probability, statistics, and statistical mechanics, and often I find it useful to perform simulations to get some sense of the underlying phenomena. Example computations include Monte Carlo, finding the eigenvalues of large random matrices (repeatedly, to get a sense of various distributional properties), and spin glass models.

I have moderate proficiency in mathematics and theoretical computer science but don't know anything about hardware or numerical computing. I apologize if this question is a bit scattershot; part of the problem is that I don't know what resources to access to learn more. (See the last point, below.)

I will be building a new computer from consumer hardware, and while it will not primarily be used to do scientific computing, I'm also curious about what decisions I could make to give it some ability in that area. For serious work, I can rent remote servers (or perhaps use my school's), but for hobby projects, it is more convenient to work locally.

The following questions are based on my very, very incomplete knowledge and research; please correct me if anything seems off.

1. For the problems I discussed above, is it more efficient (monetarily) to invest in a better CPU with many cores and support for AVX instructions (e.g. AVX512), or a dedicated discrete GPU with support for e.g. CUDA? The numpy linear algebra benchmarks I've seen indicate that the GPU route is superior for most commonly encountered operations (with some exceptions noted here). But if that's the case, what's the use of the AVX instructions?

2. It was suggested to me that I should look at "workstation" GPUs instead of "consumer" GPUs marketed toward people playing video games, with the reasoning that the former would offer better performance even at the same price point. Is this correct, and if so, why? Does the answer to this question change at all if I want to train and run neural networks?

3. Is there any good way to estimate what sufficient RAM would be? It was suggested to me that for large dimensional matrices, even 16 GB might become a bottleneck with good CPUs/GPUs.

4. Where's a good place to learn more about the hardware side of scientific computing? I've thought about picking up a computer architecture textbook, but if there are shorter, more practically oriented materials out there, I'd love to read them.

Answer 1

One issue that you should be aware of is that NVIDIA has a market segmentation strategy in which it sells relatively inexpensive GPU's to the gaming and graphics workstation markets (GeForce and Quadro) and different higher-priced models (Tesla) to the high-performance computing market.

The GPU's sold for use in gaming and graphics have limited double-precision floating-point performance compared with the models that are sold for high-performance computing. On many models, the double-precision performance is restricted to 1/32 of the single-precision performance.

To answer your specific questions:

1. The AVX instructions are of course useful in systems that don't have a powerful GPU capable of doing floating point computations. Even if you do have a GPU, it may be faster for very small computations to use the AVX vector floating point unit than to send data to the GPU, compute, and copy the results back to main memory.

2. NVIDIA's Tesla GPU's for high-performance computing are much more expensive than the gaming and graphics oriented GPU's that have limited double precision performance as explained above. For training neural networks the double-precision performance doesn't matter since single-precision floating point is normally used in training neural networks. For numerical solution of PDE's, double precision is normally used.

3. You need to think about how much memory you need in the main system and in the GPU. This will depend a lot on your particular problems, but 16 gigabytes of RAM on the main system is quite small by today's standards and you could find it limiting. The RAM on the GPU is typically smaller, which means that you often have to use block algorithms and move data on/off the GPU to solve larger problems. More RAM on the GPU is very helpful but often not absolutely necessary.

4. I'll recommend Computer Architecture: A Programmer's Perspective by Bryant and O'Halleron for its discussion of the memory hierarchy and optimization pf program performance. Another book that also discusses these issues (and covers OpenMP and MPI programming) is Introduction to High Performance Computing for Scientists and Engineers by Hager and Wellein. However, neither of these books covers GPU computing.

My advice on GPU programming is to start by making use of languages and libraries that handle the low-level details and allow you to focus on higher level issues. For example, MATLAB can make use of GPU without requiring you to do any low-level programming. At another level, the MAGMA library handles LAPACK linear algebra on the GPU with very minimal changes to your code. It's also possible to use OpenACC to write a program that can run transparently across CPU and GPU. All of these approaches keep you from having to write your own computational kernels for the GPU in CUDA or OpenCL.