Problem of multiplication of big (sparse) matrix with numpy (python)

Question

I wanted to multiply two simple (big and sparse) matrix with numpy. And I saw that the calculation fails when matrices are too big.

If i take $X$ a random vector (size $n$). With pandas, I transformed this vector with get_dummies and I obtain a matrix $M$. One can verify that $M^\top \cdot M$ is a diagonal matrix and the trace is equal to $n$. When I do it on numpy, it doesn't work for $n>2000$. The code is below :

import numpy as np
import pandas as pd
np.random.seed(11111)


n_points = 5000
X = pd.Series(np.random.uniform(low=1,high=10,size=(n_points)))
X_bis = pd.get_dummies(X.astype('int').astype('str')).values
print(np.sum(np.sum(np.matmul(X_bis.T,X_bis))))

I tried with scipy.sparse but it didn't work either. Does somebody have an idea ?

Answer 1

You are using integers that are not big enough to represent your data. See the following snippet that solves your issue.

import numpy as np
import pandas as pd

np.random.seed(11111)
n_points = 50000
X = pd.Series(np.random.randint(low=1, high=10, size=(n_points)))
X_bis = pd.get_dummies(X, dtype=np.uint64).values
print(np.sum(X_bis.T @ X_bis))