# Understanding OpenCL performance

I'm using ViennaCL's interface to Eigen as a way to leverage OpenCL. Specifically, I'm using the ::viennacl::linalg::bicgstab_tag with an Eigen sparse matrix. However, the performance isn't what I hoped that it would be.

What tools on Windows 7/Mac OS X/Linux should I use to understand the performance bottlenecks?

• Hi Joseph, welcome to scicomp! This question will be much better if we can get a few more details from you. How large are the matrices that you are trying to solve for? Are you comparing against some of the more mature libraries such as PETSc or Trilinos? How are you measuring performance right now (flop/s, mb/s?)? How many iterations is the solver taking? – Aron Ahmadia Feb 17 '12 at 19:47
• The code is in development right now, so I'll give rough sizes rather than hard upper bounds. Current data sets have sizes of 500 by 500. I'd expect that 50000 x 50000 would be the upper limit. The solvers are still another area of research. Suffice it to say that any C callable package could be used through the appropriate application of C++ templates. Currently, we are using Eigen3 or TAUCS and in another part of the code MKL's paradiso. I'm hoping to settle on one due to support issues. The iterative solvers use about 4 iterations. Performance is being measured in wall time. – Joseph Winston Feb 18 '12 at 1:50
• What are you comparing to? What hardware are you running the OpenCL code on? What compiler did you use? Why do you think $50k\times 50k$ is an "upper limit"? What physics and discretization are you solving with? – Jed Brown Feb 18 '12 at 6:52
• The targeted hardware on Linux/Windows is high end NVidia cards. My development is Quadro FX 4800 running in a HP Z800. Compiler is NVidia's CUDA 4.1. Upper limit is a guess that depends on two factors: The patience of the end user and the fineness that they want to use. \ – Joseph Winston Feb 18 '12 at 17:04

General statement: For small system sizes, there is little benefit of using OpenCL at all.

Alright, now for the justification: There is a certain amount of overhead associated with each OpenCL kernel launch. Exact timings depend on the underlying hardware, but as a rule of thumb one can use a pessimistic estimate of 10 microseconds for CPUs and 100 microseconds for GPUs as I've once reported in this thread at the Intel OpenCL forum. This is A LOT considering that modern hardware provides many GFLOPs of processing power. For example, adding two vectors with 100.000 entries each on a GPU with 100 GB/sec memory bandwidth requires 3 * 8 * 100.000 Bytes (approx. 2MB) of data to be transferred, taking 20 us. Thus, even adding up two vectors of size 100.000 can show significant kernel launch overhead. In practice, kernel launch overhead can be reduced if kernels are enqueued while another kernel is still active - this, however, requires again that kernel execution times are sufficiently large.

Since most operations inside Krylov solvers are comparable in complexity to vector additions (this often applies to sparse matrix-vector multiplications as well), benchmarks for system sizes below 10.000 by 10.000 essentially measure OpenCL kernel launch overheads only. I thus recommend to run benchmarks again with your upper limit 50k by 50k. For smaller system sizes, just use BiCGStab with the Eigen matrix directly (this is one of the reasons why ViennaCL offers generic implementations...) or give sparse direct solvers a try.

For dense systems (matrices), the notion of a 'small system' is certainly shifted to smaller values. Still, below system sizes of about 1000x1000 overheads become significant again.

• That's what I GUESSED was happening since I'm new to the whole CUDA/OpenCL world. But without experience and/or tools, that is all that it was. I had hoped to select "one size fits all" but it looks like I'll need to make performance measurements at different array sizes and then select the best place to do this work. – Joseph Winston Feb 18 '12 at 17:08
• This is why I recommend a simple performance analysis in my post where you just measure the runtime as a function of problem size. the accumulateed overhead will typically show up for small problem sizes where the performance curve will then be flat. then ones the work load increases you typically see the cpu time scale up until saturation (typically bandwidth). – Allan P. Engsig-Karup Feb 18 '12 at 17:23
• Joseph, as you've correctly experienced, CUDA/OpenCL is not the solution to all problems. Still, they are able to increase performance substantially if used correctly (and if the respective algorithm provides sufficient parallelism). I tried to give an appropriate amount of background information on when to use and when not to use CUDA/OpenCL for your setting :-) – Karl Rupp Feb 19 '12 at 7:51

A very simple means is to compute by hand the amount of useful work that are to be done (i.e. either amount of data to be transferred or the amount of useful floating point operations (flops) or both).

Then do some simple timings and average them to get an absolute prediction of performance for a given problem size (time vs. problem size) and from this you can assess overall performance characteristics. This is a simple and useful means for making comparisons with hardware specifications and/or performance of other software.

Furthermore, To do a complete performance breakdown you will need tools such as TotalView for profiling and identifying performance bottlenecks.