Memory usage within graphics cards

Question

I have a question about the memory usage within the graphics card. What does happen when the used application needs more memory than the graphic card does contain? let`s say the reserved main memory is also not enough.

The scenario I am talking about is something like:

Consider a graphics card with, let's say, 50 MB memory usable for texture information. The reserved main memory space for texture data is also around 50 MB. The application I want to run contains five data sets of each 25 MB that are needed at the same time, so you have a total of 100 MB memory, but 125 MB is needed.

Answer 1

The most common case is that you cannot allocate a memory block larger than what is physically available for your GPU card. However, depending on your hardware and driver, the maximum allocatable size might be a fraction of the total physically installed memory (e.g. old AMD opencl drivers only allowed a maximum of 500 MB), so it is better if you read into the specs of the vendor (NVIDIA, AMD, or Intel).

In the case that a GPU kernel of your implementation needs to access more of the global, or local / shared memory than what one can allocate at once, then you have to manually manage the memory by allocating and freeing memory segments (smaller than maximum memory) and transfer them between host and GPU as needed.

As far as my experience goes, memory management is crucial for achieving high performance for GPU applications. The reason for this is that for most applications, transferring data through the PCIe between host and GPU can become a big bottleneck. To illustrate the PCIe problem, consider that the maximum bandwidth of PCIe 2.0 (x16 lanes) is 8 GB/s bidirectional, whereas the bandwidth from on board GPU memory to GPU chip is +140 GB/s. Moreover, all the tiny cores of a GPU can collectively process several terabytes of data per second, thus most high performance GPU kernels also need to make efficient use of shared / local and register memory (which is orders of magnitude faster than accessing global memory).

An alternative to explicit memory management, is to use extensions to opencl where a GPU kernel can directly access, from the discrete GPU chip, memory segments that reside on host (as fas as I know, AMD allows this). In such a case, the GPU would halt execution until data is received from host, these can be a very long time from the perspective of the GPU thread.

A more interesting scenario happens when the CPU and GPU are integrated in the same chip. In such a case, CPU and GPU share the host memory (consider the fusion architecture of AMD) so there should not be necessary to copy data between host and device.

Answer 2

On NVIDIA cards at least, it is possible to allocate pinned memory on the CPU for use by the graphics card. Pointers to this block of memory are in the GPU's global memory space (so they can be accessed by your kernels) but reads/writes to them must still go through the PCI-E bus. The drawback is that you cannot use too much of this memory or your OS will become very angry with you. Is this what you meant by "reserved main memory"?

It's also possible to handle compute/memory transfer at once, so if an application doesn't need all the data at once you can have the kernel process a slice of data while the next batch is transferring, do something with the result, and then continue processing and transferring.

Edit: referring to your specific example, it may also be possible to store 50 kB in texture memory, 50 kB in global (I assume this is what you meant by main?), and then find somewhere else to squeeze the last 25kB in, like constant memory. You would only be able to read these last 25 kB, so this may not work in all cases. Another option would be to use the pinned memory, as I discussed earlier. The advantage of constant memory is that the data is actually on the device, but the read-only restriction may be too onerous for some applications.