Why not give GMPY2 a try? From the introduction:
gmpy2 is a C-coded Python extension module that supports multiple-precision arithmetic. gmpy2 is the successor to the original gmpy module. The gmpy module only supported the GMP multiple-precision library. gmpy2 adds support for the MPFR (correctly rounded real floating-point arithmetic) and MPC (correctly rounded complex floating-point arithmetic) libraries. gmpy2 also updates the API and naming conventions to be more consistent and support the additional functionality.
GMPY2 supports the integer arithmetic I assume you need and I doubt you can find anything faster.
On *nix based systems, you'll need to make sure that the latest version of GMP is installed (GNU MP Bignum Library. On Windows, you can find a Python wheel courtesy of Christophe Gohlke here.