# Problem of multiplication of big (sparse) matrix with numpy (python)

I wanted to multiply two simple (big and sparse) matrix with numpy. And I saw that the calculation fails when matrices are too big.

If i take $$X$$ a random vector (size $$n$$). With pandas, I transformed this vector with get_dummies and I obtain a matrix $$M$$. One can verify that $$M^\top \cdot M$$ is a diagonal matrix and the trace is equal to $$n$$. When I do it on numpy, it doesn't work for $$n>2000$$. The code is below :

import numpy as np
import pandas as pd
np.random.seed(11111)

n_points = 5000
X = pd.Series(np.random.uniform(low=1,high=10,size=(n_points)))
X_bis = pd.get_dummies(X.astype('int').astype('str')).values
print(np.sum(np.sum(np.matmul(X_bis.T,X_bis))))


I tried with scipy.sparse but it didn't work either. Does somebody have an idea ?

• I am not able to reproduce any error, even for n_points = 1000000. – nicoguaro Apr 29 '19 at 16:55
• "It didn't work" / "the calculation fails" are not very helpful sentences; it's like going to the doctor and saying only "I'm not feeling well". What are your symptoms? Do you get an error message? Could you add more detail? – Federico Poloni Apr 29 '19 at 22:37
• When I check: np.sum(np.sum(np.matmul(X_bis.T, X_bis))) == n_points, for n_points <2000, I get True and False if n_points> 2000 (I tried on Jupyter or directly on a terminal, on two different computers, and I get the same problem) – Thomas Delattre Apr 30 '19 at 7:33

## 1 Answer

You are using integers that are not big enough to represent your data. See the following snippet that solves your issue.

import numpy as np
import pandas as pd

np.random.seed(11111)
n_points = 50000
X = pd.Series(np.random.randint(low=1, high=10, size=(n_points)))
X_bis = pd.get_dummies(X, dtype=np.uint64).values
print(np.sum(X_bis.T @ X_bis))

• Thank you for your answer. As proposed, by changing the type of the matrix, it works well – Thomas Delattre May 1 '19 at 12:11