# Problem of multiplication of big (sparse) matrix with numpy (python)

I wanted to multiply two simple (big and sparse) matrix with numpy. And I saw that the calculation fails when matrices are too big.

If i take $$X$$ a random vector (size $$n$$). With pandas, I transformed this vector with get_dummies and I obtain a matrix $$M$$. One can verify that $$M^\top \cdot M$$ is a diagonal matrix and the trace is equal to $$n$$. When I do it on numpy, it doesn't work for $$n>2000$$. The code is below :

import numpy as np
import pandas as pd
np.random.seed(11111)

n_points = 5000
X = pd.Series(np.random.uniform(low=1,high=10,size=(n_points)))
X_bis = pd.get_dummies(X.astype('int').astype('str')).values
print(np.sum(np.sum(np.matmul(X_bis.T,X_bis))))


I tried with scipy.sparse but it didn't work either. Does somebody have an idea ?

• I am not able to reproduce any error, even for n_points = 1000000. Apr 29, 2019 at 16:55
• "It didn't work" / "the calculation fails" are not very helpful sentences; it's like going to the doctor and saying only "I'm not feeling well". What are your symptoms? Do you get an error message? Could you add more detail? Apr 29, 2019 at 22:37
• When I check: np.sum(np.sum(np.matmul(X_bis.T, X_bis))) == n_points, for n_points <2000, I get True and False if n_points> 2000 (I tried on Jupyter or directly on a terminal, on two different computers, and I get the same problem) Apr 30, 2019 at 7:33

import numpy as np