I'm interested in problems around probability, statistics, and statistical mechanics, and often I find it useful to perform simulations to get some sense of the underlying phenomena. Example computations include Monte Carlo, finding the eigenvalues of large random matrices (repeatedly, to get a sense of various distributional properties), and spin glass models.
I have moderate proficiency in mathematics and theoretical computer science but don't know anything about hardware or numerical computing. I apologize if this question is a bit scattershot; part of the problem is that I don't know what resources to access to learn more. (See the last point, below.)
I will be building a new computer from consumer hardware, and while it will not primarily be used to do scientific computing, I'm also curious about what decisions I could make to give it some ability in that area. For serious work, I can rent remote servers (or perhaps use my school's), but for hobby projects, it is more convenient to work locally.
The following questions are based on my very, very incomplete knowledge and research; please correct me if anything seems off.
For the problems I discussed above, is it more efficient (monetarily) to invest in a better CPU with many cores and support for AVX instructions (e.g. AVX512), or a dedicated discrete GPU with support for e.g. CUDA? The numpy linear algebra benchmarks I've seen indicate that the GPU route is superior for most commonly encountered operations (with some exceptions noted here). But if that's the case, what's the use of the AVX instructions?
It was suggested to me that I should look at "workstation" GPUs instead of "consumer" GPUs marketed toward people playing video games, with the reasoning that the former would offer better performance even at the same price point. Is this correct, and if so, why? Does the answer to this question change at all if I want to train and run neural networks?
Is there any good way to estimate what sufficient RAM would be? It was suggested to me that for large dimensional matrices, even 16 GB might become a bottleneck with good CPUs/GPUs.
Where's a good place to learn more about the hardware side of scientific computing? I've thought about picking up a computer architecture textbook, but if there are shorter, more practically oriented materials out there, I'd love to read them.