2
$\begingroup$

Let's define scaling, as how linear the speedup of using more than one GPU or CPU is. For example, having 2 GPUs gives you 2x faster execution time.

I have noticed that in many software (e.g. Molecular Dynamics, Gromacs, AMBER, OPENMM), it scales very badly for more than 2 GPUs, but it is not a problem for CPUs (you can use hundreds of CPUs, even located on different nodes). What is stopping GPUs from achieving the same scaling as CPUs? Why CPUs even located physically far away with huge latency, can achieve better scaling than GPUs?

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ If you look at an MD software (HOOMD-blue) designed for GPUs from the very start, the observation does not seem to apply. Jens Glaser, et. al. "Strong scaling of general-purpose molecular dynamics simulations on GPUs," Computer Physics Communications 192 (2015): 97-107: " We show that strong scaling speed-ups in excess of 50x are attainable on the Titan supercomputer, and weak scaling holds over three orders of magnitude in system size. " $\endgroup$
    – njuffa
    Feb 10 at 21:11
  • $\begingroup$ You might also be interested in the molecular-dynamics, high-performance-computing, and graphical-processing-units tags at MMSE. There's also specific tags for AMBER and Gromacs there! Finally, please come and say "hello" in the Gromacs chat room so that we remember that you're interested in Gromcas! $\endgroup$
    – Nike Dattani
    Feb 11 at 19:09
  • 2
    $\begingroup$ You might have additional communication channels. Have to move memory from GPU to RAM through network to RAM and then GPU versus just RAM-network-RAM. This gets more challenging if the GPUs are contending for link bandwidth. Just a hypothesis. $\endgroup$
    – Richard
    Feb 11 at 22:02
  • $\begingroup$ Too much missing information in this question. You "have noticed" on what platform? How are your "more than 2 GPUs" arranged? On a single node? In that case you probably have a bandwidth problem. $\endgroup$
    – Victor Eijkhout
    Feb 12 at 16:40
  • 1
    $\begingroup$ Too many unknowns in the question (as pointed out above) to answer this fully. One thing worth noticing is that a shiny new A100 will net you 10 TFLOPS DP, but a modern Intel Xeon will get you only ~0.05 TFLOPS (200x difference). At the same time, these compute units are paired with a (somewhat) similar amount of memory. So, for a similarly-sized problem, one spends less time computing on the GPU-based machine but a similar amount of time communicating (network). $\endgroup$
    – user20857
    Feb 14 at 4:31

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.