I am writing a program that is supposed to use multiple GPUs on a single node using CUDA Fortran. Although I've looked through the Portland Group CUDA Fortran Reference, I am still unclear about how to make memory allocation work in my case.
I am trying to split a simulation domain between multiple GPUs, such that the GPUs share the load. I have a few constraints:
The field may not be perfectly evenly split, due to partitioning algorithms in use. So a given GPU device usually will end up with a slightly different array than its fellow devices.
The code has to be scalable. Some machines may only have a single GPU available, while other machines may have two, three, our four GPUs, based on PCI ports, etc.
The field is large enough that I would rather not have simple copies of the field variables, because that would limit my field size to 1/N of the ideal scenario, where N is the number of GPUs attached to the node.
I tried to follow the procedure I would use if implementing in MPI, given the assumption that each GPU keeps a separate symbol table (distributed memory assumption), and thus can be allocated individually, and used an approach similar to this:
program multiGPU
implicit none
doubleprecision, allocatable, device :: fieldVAR(:)
integer, parameter :: NGPUS=3 ! For the sake of this example
integer :: istat, i, fieldsizes(NGPUS)
! field size assigned to each GPU, which would be done through
! some partitioning algorithm. Arbitrary assignment here:
fieldsizes(1)=300
fieldsizes(2)=400
fieldsizes(3)=350
do i=1, NGPUS
istat= cudaSetDevice(i-1)
allocate(fieldVAR(fieldsizes(i)))
!
! Assign initial values etc.
!
end do
end program multiGPU
Because in MPI each process keeps a separate symbol table, and the only thing common to all processes are the environment variables such as the ones with the MPI_* prefix However, this tactic does not seem to work as the program seems to go with the latest allocation done, and thus suggests that even though there are two separate units they keep a single symbol table, probably due to having the same host code.
I am wondering if there is a better way to split the simulation domain than hard coding "GPU1_fieldvar" and "GPU2_fieldvar".