I wanted to multiply two simple (big and sparse) matrix with numpy. And I saw that the calculation fails when matrices are too big.
If i take $X$ a random vector (size $n$). With pandas, I transformed this vector with
get_dummies and I obtain a matrix $M$. One can verify that $M^\top \cdot M$ is a diagonal matrix and the trace is equal to $n$. When I do it on numpy, it doesn't work for $n>2000$. The code is below :
import numpy as np import pandas as pd np.random.seed(11111) n_points = 5000 X = pd.Series(np.random.uniform(low=1,high=10,size=(n_points))) X_bis = pd.get_dummies(X.astype('int').astype('str')).values print(np.sum(np.sum(np.matmul(X_bis.T,X_bis))))
I tried with
scipy.sparse but it didn't work either. Does somebody have an idea ?