2
$\begingroup$

I wanted to multiply two simple (big and sparse) matrix with numpy. And I saw that the calculation fails when matrices are too big.

If i take $X$ a random vector (size $n$). With pandas, I transformed this vector with get_dummies and I obtain a matrix $M$. One can verify that $M^\top \cdot M$ is a diagonal matrix and the trace is equal to $n$. When I do it on numpy, it doesn't work for $n>2000$. The code is below :

import numpy as np
import pandas as pd
np.random.seed(11111)


n_points = 5000
X = pd.Series(np.random.uniform(low=1,high=10,size=(n_points)))
X_bis = pd.get_dummies(X.astype('int').astype('str')).values
print(np.sum(np.sum(np.matmul(X_bis.T,X_bis))))

I tried with scipy.sparse but it didn't work either. Does somebody have an idea ?

$\endgroup$
3
  • $\begingroup$ I am not able to reproduce any error, even for n_points = 1000000. $\endgroup$
    – nicoguaro
    Apr 29, 2019 at 16:55
  • 1
    $\begingroup$ "It didn't work" / "the calculation fails" are not very helpful sentences; it's like going to the doctor and saying only "I'm not feeling well". What are your symptoms? Do you get an error message? Could you add more detail? $\endgroup$ Apr 29, 2019 at 22:37
  • $\begingroup$ When I check: np.sum(np.sum(np.matmul(X_bis.T, X_bis))) == n_points, for n_points <2000, I get True and False if n_points> 2000 (I tried on Jupyter or directly on a terminal, on two different computers, and I get the same problem) $\endgroup$ Apr 30, 2019 at 7:33

1 Answer 1

1
$\begingroup$

You are using integers that are not big enough to represent your data. See the following snippet that solves your issue.

import numpy as np
import pandas as pd

np.random.seed(11111)
n_points = 50000
X = pd.Series(np.random.randint(low=1, high=10, size=(n_points)))
X_bis = pd.get_dummies(X, dtype=np.uint64).values
print(np.sum(X_bis.T @ X_bis))
$\endgroup$
1
  • $\begingroup$ Thank you for your answer. As proposed, by changing the type of the matrix, it works well $\endgroup$ May 1, 2019 at 12:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.