# Problem of multiplication of big (sparse) matrix with numpy (python)

I wanted to multiply two simple (big and sparse) matrix with numpy. And I saw that the calculation fails when matrices are too big.

If i take $$X$$ a random vector (size $$n$$). With pandas, I transformed this vector with get_dummies and I obtain a matrix $$M$$. One can verify that $$M^\top \cdot M$$ is a diagonal matrix and the trace is equal to $$n$$. When I do it on numpy, it doesn't work for $$n>2000$$. The code is below :

import numpy as np
import pandas as pd
np.random.seed(11111)

n_points = 5000
X = pd.Series(np.random.uniform(low=1,high=10,size=(n_points)))
X_bis = pd.get_dummies(X.astype('int').astype('str')).values
print(np.sum(np.sum(np.matmul(X_bis.T,X_bis))))


I tried with scipy.sparse but it didn't work either. Does somebody have an idea ?

• I am not able to reproduce any error, even for n_points = 1000000. – nicoguaro Apr 29 '19 at 16:55
• "It didn't work" / "the calculation fails" are not very helpful sentences; it's like going to the doctor and saying only "I'm not feeling well". What are your symptoms? Do you get an error message? Could you add more detail? – Federico Poloni Apr 29 '19 at 22:37
• When I check: np.sum(np.sum(np.matmul(X_bis.T, X_bis))) == n_points, for n_points <2000, I get True and False if n_points> 2000 (I tried on Jupyter or directly on a terminal, on two different computers, and I get the same problem) – Thomas Delattre Apr 30 '19 at 7:33

import numpy as np