2
$\begingroup$

I wanted to multiply two simple (big and sparse) matrix with numpy. And I saw that the calculation fails when matrices are too big.

If i take $X$ a random vector (size $n$). With pandas, I transformed this vector with get_dummies and I obtain a matrix $M$. One can verify that $M^\top \cdot M$ is a diagonal matrix and the trace is equal to $n$. When I do it on numpy, it doesn't work for $n>2000$. The code is below :

import numpy as np
import pandas as pd
np.random.seed(11111)


n_points = 5000
X = pd.Series(np.random.uniform(low=1,high=10,size=(n_points)))
X_bis = pd.get_dummies(X.astype('int').astype('str')).values
print(np.sum(np.sum(np.matmul(X_bis.T,X_bis))))

I tried with scipy.sparse but it didn't work either. Does somebody have an idea ?

$\endgroup$
  • $\begingroup$ I am not able to reproduce any error, even for n_points = 1000000. $\endgroup$ – nicoguaro Apr 29 at 16:55
  • 1
    $\begingroup$ "It didn't work" / "the calculation fails" are not very helpful sentences; it's like going to the doctor and saying only "I'm not feeling well". What are your symptoms? Do you get an error message? Could you add more detail? $\endgroup$ – Federico Poloni Apr 29 at 22:37
  • $\begingroup$ When I check: np.sum(np.sum(np.matmul(X_bis.T, X_bis))) == n_points, for n_points <2000, I get True and False if n_points> 2000 (I tried on Jupyter or directly on a terminal, on two different computers, and I get the same problem) $\endgroup$ – Thomas Delattre Apr 30 at 7:33
1
$\begingroup$

You are using integers that are not big enough to represent your data. See the following snippet that solves your issue.

import numpy as np
import pandas as pd

np.random.seed(11111)
n_points = 50000
X = pd.Series(np.random.randint(low=1, high=10, size=(n_points)))
X_bis = pd.get_dummies(X, dtype=np.uint64).values
print(np.sum(X_bis.T @ X_bis))
$\endgroup$
  • $\begingroup$ Thank you for your answer. As proposed, by changing the type of the matrix, it works well $\endgroup$ – Thomas Delattre May 1 at 12:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.