Eigen 3 is a nice C++ template library some of whose routines are parallelized.
c.f. Eigen documentation
The parallelization is OMP only, so if you intend to parallelise using MPI (and OMP) it is probably not suitable for your purpose.
The nice feature of Eigen is that you can swap in a high performance BLAS library (like MKL or OpenBLAS) for some routines by simply using
#define EIGEN_USE_BLAS (and other macros).
Similarly Armadillo allows for node-level parallelism only. In my experience it is better to use Eigen since it is easier to interface with the raw C++ arrays in Eigen, which facilitates use of other libraries (e.g. ARPACK++).
In my experience I would advise against using GSL for linear algebra. I have found its performance to be lacking and the usability to be worse than that of Eigen.
If you plan to execute linear solvers (e.g. BiCGstab) on multiple nodes I would advise you to use Trilinos. I have used it in my research codes with fairly little delving into its documentation due to the good examples available on the Trilinos homepage. Furthermore its performance is decent and can be fine-tuned by including good BLAS/LAPACK libraries during the compilation.
Similar should hold for PETSc, although I have never actively used its LA routines. In my experience PETSc is dependency hell, if you want to use performance-optimized (e.g. optimized for the CPU architecture you're using) versions of the libraries it requires. The performance should be fairly decent, too, I think, since PETSc relies on common LA libraries (BLAS, LAPACK, SCALAPACK etc.).
Long story short: For interoperability and good performance on a single node (using OpenMP) I advise to use Eigen with OpenBLAS. If you want to use multiple nodes via MPI and let the library figure out how to solve a system using multiple nodes then use Trilinos.