I apologise if this is somewhat of a rookie question. So, from my understanding, on a GPU board, far more of the space is allocated to ALUs compared to CPUs which have far more cache available. This should mean that a typical GPU is capable of higher FLOP counts. This I understood to be the advantage of a GPU (in a nutshell). However, looking at the numbers for some of the state of the art processors on the market, Nvidia's 3090 card outputs around 0.5 TFLOPS (double precision) at \$1499, while Intel's Xeon 3175-X processor outputs 1.5 TFLOPS (also double precision I presume) at \$2999. So the CPU is providing higher double precision FLOP count per dollar.
I find these figures to be a bit confusing. What exactly is the advantage of the GPU if not in overall (or effective) FLOP counts? Are GPUs only advantageous when performing single or half precision operations?
source for 3090 FLOP counts: https://en.wikipedia.org/wiki/GeForce_30_series
source for Xeon FLOP counts: https://www.intel.com/content/dam/support/us/en/documents/processors/APP-for-Intel-Xeon-Processors.pdf