CUDA Fortran: Multi GPU Programming and memory allocation

Question

I am writing a program that is supposed to use multiple GPUs on a single node using CUDA Fortran. Although I've looked through the Portland Group CUDA Fortran Reference, I am still unclear about how to make memory allocation work in my case.

I am trying to split a simulation domain between multiple GPUs, such that the GPUs share the load. I have a few constraints:

• The field may not be perfectly evenly split, due to partitioning algorithms in use. So a given GPU device usually will end up with a slightly different array than its fellow devices.

• The code has to be scalable. Some machines may only have a single GPU available, while other machines may have two, three, our four GPUs, based on PCI ports, etc.

• The field is large enough that I would rather not have simple copies of the field variables, because that would limit my field size to 1/N of the ideal scenario, where N is the number of GPUs attached to the node.

I tried to follow the procedure I would use if implementing in MPI, given the assumption that each GPU keeps a separate symbol table (distributed memory assumption), and thus can be allocated individually, and used an approach similar to this:

  program multiGPU
  implicit none
  doubleprecision, allocatable, device :: fieldVAR(:)
  integer, parameter :: NGPUS=3 ! For the sake of this example
  integer :: istat, i, fieldsizes(NGPUS)

  ! field size assigned to each GPU, which would be done through
  ! some partitioning algorithm. Arbitrary assignment here:
  fieldsizes(1)=300
  fieldsizes(2)=400
  fieldsizes(3)=350

  do i=1, NGPUS
      istat= cudaSetDevice(i-1)
      allocate(fieldVAR(fieldsizes(i)))
      !
      !  Assign initial values etc.
      !
  end do
  end program multiGPU

Because in MPI each process keeps a separate symbol table, and the only thing common to all processes are the environment variables such as the ones with the MPI_* prefix However, this tactic does not seem to work as the program seems to go with the latest allocation done, and thus suggests that even though there are two separate units they keep a single symbol table, probably due to having the same host code.

I am wondering if there is a better way to split the simulation domain than hard coding "GPU1_fieldvar" and "GPU2_fieldvar".

Answer 1

I'm not sure if this would be a better fit for the StackOverflow, but here goes.

The best way to do this is to make a new type, which contains the allocatable array for that particular GPU.

Have a look at this article from the Portland Group, describing how to do multi-GPU computations with CUDA Fortran.