Why is SIMD programming on such a low level in CPU's compared to GPU's?

Question

Just speaking from my experience in c++ - Its really tricky to take advantage of the SIMD capabilities of modern CPU's for more complex algorithms. If I see an opportunity from a high level OO perspective, I would then have to break the whole thing down to a very non-OO kind of code, babysitting every register individually even if they have no interaction with each other and not only the part that benefits from vectorizing but often the entire flow of data along with data formats have to be redesigned into a more basic and harder to manage form.

The automatic vectorization tools(qvec/openmp) offer no benefit either since they can only deal with just the most basic data types and loops with no OO concepts supported at all and imo only succeed in obscuring the code and limiting your options over the manual way.

Meanwhile, when coding for GPGPU's, you essentially get to do it in a perfectly OO manner(more/less, depending on the language) while focusing on the journey of one piece of data through the pipeline which just gets duplicated to every other piece of data, brilliant!

I know that gpu "cores" aren't the same as a SIMD unit on a cpu but from what I've gathered they are much closer to that than actual individual cores since they have to move in a kind of a lockstep.

So basically, why is that?

Answer 1

Before I try to answer your question let me comment on the words "low level" in your question statement. In my opinion, I prefer not to say one programming model(one of SIMD and SIMT) is at a low level. As an example, during my undergraduage diploma project I had experienced many low level issues about GPU programming, which I have not encountered in CPU programming. Most of these issues involve the memory architecture of GPU, such as memory coalescing access, bank conflict, etc. You can search these topics to get a feeling of low-level-ness from GPU programming.

And now return to your question, by "low level" I assume you mean that you have to refactor your code or even redesign your algorithm to fit the SIMD model, while on GPU(e.g. using CUDA) it may not require too much work like that. In some cases your source code for CPU can even compiled without any errors using CUDA compiler. But to vectorize a serial program in SIMD you have to translate all branching code(if, while block).

(I am not quite sure of what you mean by "OOP" in your question statement. Please tell me if I made a wrong assumption.)

The reason is that GPU programming(CUDA) uses a different model from the one used in CPU programming. The model for GPU programming is called SIMT. T is for thread.

In SIMD model, all arithmetic operations have to performed in a synchronized way. So any branching execution is not allowed. While in CUDA, branching is allowed in term of CUDA syntax, and more importantly, it is supported by hardware architecture and CUDA runtime. When a branching execution happens in CUDA/SIMT, the thread manager will coordinate the execution: those threads with identical execution path will be executed, and the diverse threads will be queued for later execution. During this process, other independent threads may be brought in to keep a high throughput.

As you can see, the SIMT model helps you to handle the branching execution.

Finally, I would like to let you know that you can programming SIMD in a "high level" by introducing another syntax layer. You can try https://ispc.github.io/.

Answer 2

I think you are comparing using high-level libraries built on GPGPU techniques, like C++ AMP, to programming SIMD at the lowest possible assembly language or intrinsics level. That's not a fair comparison.

Since you specifically mentioned C++ AMP, let me use a basic example to argue that your premise isn't true. The introductory AMP example shows how to parallelize the following simple vector addition using AMP:

#include <iostream>

void StandardMethod() {

    int aCPP[] = {1, 2, 3, 4, 5};
    int bCPP[] = {6, 7, 8, 9, 10};
    int sumCPP[5];

    for (int idx = 0; idx < 5; idx++)
    {
        sumCPP[idx] = aCPP[idx] + bCPP[idx];
    }

    for (int idx = 0; idx < 5; idx++)
    {
        std::cout << sumCPP[idx] << "\n";
    }
}

To do this, you have to use an AMP library function and write your loop in a specific way. What if we instead want to parallelize this loop using SIMD instructions on the CPU? We only have to compile it with a reasonably modern compiler and suitable flags!

Compiling the above code with gcc 4.8.3 on Linux and disassembling it, I obtain

$ gcc -O3 -c vectest.cpp && objdump -M intel -d vectest.o 

vectest.o:     file format elf64-x86-64


Disassembly of section .text:

0000000000000000 <_Z14StandardMethodv>:
   0:   55                      push   rbp
   1:   53                      push   rbx
   2:   48 83 ec 68             sub    rsp,0x68
   6:   c7 04 24 01 00 00 00    mov    DWORD PTR [rsp],0x1
   d:   c7 44 24 04 02 00 00    mov    DWORD PTR [rsp+0x4],0x2
  14:   00 
  15:   48 8d 5c 24 40          lea    rbx,[rsp+0x40]
  1a:   c7 44 24 08 03 00 00    mov    DWORD PTR [rsp+0x8],0x3
  21:   00 
  22:   c7 44 24 0c 04 00 00    mov    DWORD PTR [rsp+0xc],0x4
  29:   00 
  2a:   48 8d 6c 24 54          lea    rbp,[rsp+0x54]
  2f:   66 0f 6f 04 24          movdqa xmm0,XMMWORD PTR [rsp]
  34:   c7 44 24 20 06 00 00    mov    DWORD PTR [rsp+0x20],0x6
  3b:   00 
  3c:   c7 44 24 24 07 00 00    mov    DWORD PTR [rsp+0x24],0x7
  43:   00 
  44:   c7 44 24 28 08 00 00    mov    DWORD PTR [rsp+0x28],0x8
  4b:   00 
  4c:   c7 44 24 2c 09 00 00    mov    DWORD PTR [rsp+0x2c],0x9
  53:   00 
  54:   66 0f fe 44 24 20       paddd  xmm0,XMMWORD PTR [rsp+0x20]
  5a:   66 0f 7f 44 24 40       movdqa XMMWORD PTR [rsp+0x40],xmm0
  60:   c7 44 24 10 05 00 00    mov    DWORD PTR [rsp+0x10],0x5
  67:   00 
  68:   c7 44 24 30 0a 00 00    mov    DWORD PTR [rsp+0x30],0xa
  6f:   00 
  70:   c7 44 24 50 0f 00 00    mov    DWORD PTR [rsp+0x50],0xf
  77:   00 
  78:   8b 33                   mov    esi,DWORD PTR [rbx]
  7a:   bf 00 00 00 00          mov    edi,0x0
  7f:   48 83 c3 04             add    rbx,0x4
  83:   e8 00 00 00 00          call   88 <_Z14StandardMethodv+0x88>
  88:   ba 01 00 00 00          mov    edx,0x1
  8d:   be 00 00 00 00          mov    esi,0x0
  92:   48 89 c7                mov    rdi,rax
  95:   e8 00 00 00 00          call   9a <_Z14StandardMethodv+0x9a>
  9a:   48 39 eb                cmp    rbx,rbp
  9d:   75 d9                   jne    78 <_Z14StandardMethodv+0x78>
  9f:   48 83 c4 68             add    rsp,0x68
  a3:   5b                      pop    rbx
  a4:   5d                      pop    rbp
  a5:   c3                      ret    

Disassembly of section .text.startup:

0000000000000000 <_GLOBAL__sub_I__Z14StandardMethodv>:
   0:   48 83 ec 08             sub    rsp,0x8
   4:   bf 00 00 00 00          mov    edi,0x0
   9:   e8 00 00 00 00          call   e <_GLOBAL__sub_I__Z14StandardMethodv+0xe>
   e:   ba 00 00 00 00          mov    edx,0x0
  13:   be 00 00 00 00          mov    esi,0x0
  18:   bf 00 00 00 00          mov    edi,0x0
  1d:   48 83 c4 08             add    rsp,0x8
  21:   e9 00 00 00 00          jmp    26 <_GLOBAL__sub_I__Z14StandardMethodv+0x26>

As you can see, the compiler has automatically used MMX instructions to optimize your loop, and without having to add any CPU- or library-specific annotations. So I'd say that your claim that using SIMD is more difficult than using GPGPU techniques in high-level code is not true -- quite the contrary.