Problem: I have a matrix Q of shape [51200 rows x 51200 cols] stored in a binary file, each of the element in this matrix has a data type of complex64. To load the data into memory I therefore need ~20GB of RAM, checked from
getsizeof(Q). I do have access to a server with 120GB of RAM in a LINUX machine.
My aim is to decompose the matrix with SVD.
The easiest way in Python to do this is by using
np.linalg.svd(Q). To do this, I first use
np.fromfile()to load the Q, and then execute the svd function. The problem here is, I do not know, how much memory I exactly need to compute this function. And I do get a warning
init_zgesdd failed init. Though this does not stop the computation, but in the end, the values for U,S,V* are all zeros. As checked this warning is due to memory allocation.
A second approach I tried is by using
scipy.sparse.linalg.svdslibrary. Since there are a lot of zeros (about 20%), I thought defining the matrix as sparse would have better memory usage. I found that while running this, the consumption of memory fluctuates from 50GB to 100GB, but it gets killed after running about 15-20 min.
I have also looked into on how can I decrease the precision of the matrix element. As of now I am using complex64 (that is 32 bit float for each real and imag part), I am not seeing any option for making it complex32.
I wanted to know the best way to compute SVD for such matrix.