For small system sizes, there is little benefit of using OpenCL at all.
Alright, now for the justification: There is a certain amount of overhead associated with each OpenCL kernel launch. Exact timings depend on the underlying hardware, but as a rule of thumb one can use a pessimistic estimate of 10 microseconds for CPUs and 100 microseconds for GPUs as I've once reported in this thread at the Intel OpenCL forum. This is A LOT considering that modern hardware provides many GFLOPs of processing power. For example, adding two vectors with 100.000 entries each on a GPU with 100 GB/sec memory bandwidth requires 3 * 8 * 100.000 Bytes (approx. 2MB) of data to be transferred, taking 20 us. Thus, even adding up two vectors of size 100.000 can show significant kernel launch overhead. In practice, kernel launch overhead can be reduced if kernels are enqueued while another kernel is still active - this, however, requires again that kernel execution times are sufficiently large.
Since most operations inside Krylov solvers are comparable in complexity to vector additions (this often applies to sparse matrix-vector multiplications as well), benchmarks for system sizes below 10.000 by 10.000 essentially measure OpenCL kernel launch overheads only. I thus recommend to run benchmarks again with your upper limit 50k by 50k. For smaller system sizes, just use BiCGStab with the Eigen matrix directly (this is one of the reasons why ViennaCL offers generic implementations...) or give sparse direct solvers a try.
For dense systems (matrices), the notion of a 'small system' is certainly shifted to smaller values. Still, below system sizes of about 1000x1000 overheads become significant again.