2
$\begingroup$

I wanted to compare the speeds of the GMRES implementations in the CUSP and the PETSc libraries.

The matrix (A) used for testing was a 3d Laplacian matrix obtained by using the 7 point stencil on a 20x20x20 grid => the matrix is of size 8000 x 8000. This is a banded matrix with 7 diagonals.

The RHS (b) was obtained by multiplying A by a vector of 1's. I wanted to solve Ax=b by GMRES

For both PETSC and CUSP I timed only the time taken for 200 iterations of the GMRES solver (ie time taken for reading in the matrix is not counted)

On Petsc the timings were done with PetscGetTime() and on Cusp with Cuda events.

%--------------------------

Timings

Petsc (1 cpu) took 0.0558379 sec to do 200 iterations of GMRES.

Cusp took 0.1183 sec to do the same.

%----------------------------

As you can see Petsc is 2.14 times faster than the Cusp library even on a single processor. The speed gap gets larger (obviously) on using more than 1 processor.

This experiment was done on a GTX 570 card on an Ubuntu 10.10 running CUDA 4.0 having cusp v 0.3.1 and thrust 1.6.0

I have read this paper by one of the authors of CUSP. However, they have not compared the performance of CUSP with Petsc in it.

The paper also says that the data-structure used for storing the sparse matrix matters. Accordingly they have provided several formats like DIA, ELL, HYB, COO etc. but even after I tried them all, the Cusp GMRES performance does not change and still takes 0.1183 seconds for the 200 iterations.

Here is the (quite-short) GPU code on pastebin I used while using CUSP. I am not sure if I am using the Cusp library in an optimal manner.

Although it is possible PETSc is still genuinely better than CUSP I want to know if significant speedup is possible with CUSP if used properly.

If more information is needed to answer this question please let me know. Thank you.

Improve this question
$\endgroup$
3
  • $\begingroup$ What's the host processor? What controls how many GPU threads you were using? $\endgroup$ – Bill Barth Feb 26 '13 at 22:30
  • $\begingroup$ The host processor is an intel Core i7 960. The CUSP library itself controls the number of threads launched, although I don't know if there is a lower level interface which gives the user over such details. $\endgroup$ – smilingbuddha Feb 26 '13 at 23:20
  • 1
    $\begingroup$ Hrm. One concern I have is that there may be some startup overheads that are killing your performance. This problem is pretty small (8k rows * (7+2) elements per row (and vector) * 8 bytes/double is only 500MB). I wonder if there's not enough work here to really warm up the GPU. Also, it could be that CUSP is lazy (not in a bad way) about when it moves data, starts threads, etc. You've done the effort to try to only time the iteration, but maybe you've actually timed more than you think if it waited to move the data or start the threads (or thought the problem too small to use the whole GPU). $\endgroup$ – Bill Barth Feb 26 '13 at 23:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.