Time and memory required to diagonalize a 18000 by 18000 matrix using numpy in python

Question

Can someone give an estimate of the Time and memory required to diagonalize a 20000 by 20000 complex hermitian matrix using numpy in python ?

Answer 1

A 20000 by 20000 double-precision complex matrix requires

$20000 \times 20000 \times 8 \times 2=6.4 \mbox{gigabytes}$

of RAM. The LAPACK routines ZHEEV that will do the work for you will store the result back into that array. ZHEEV requires some additional space for the eigenvalue outputs and some workspace, but these are negligible in comparison with the size of the original matrix. 8 gigabytes of RAM should be sufficient if you're careful to avoid unnecessary storage allocations. 16 gigabytes should do it if you need to keep a copy of the original matrix too.

As for time, this will depend heavily on what processor you're using and what implementation of the BLAS/LAPACK libraries you're using. However, on my desktop machine using 10 cores, with the Intel MKL version of the BLAS/LAPACK routine, diagonalizing a random 20,000 by 20,000 Hermitian complex matrix took 522 seconds (a bit less than 10 minutes.)

You should check to make sure that you're using an optimized BLAS/LAPACK library to get the best performance out of your hardware. It's likely that the distribution of Python you're using does not include optimized BLAS/LAPACK libraries. If you want to stick with open-source code then Openblas is probably your best bet. If you're willing to deal with Intel's licensing, the Intel Distribution for Python would be a good choice.

Once you've got a reasonably optimized BLAS/LAPACK library setup, you could experiment with smaller test cases and check the O(n^3) scaling to predict how long the 20,000 by 20,000 case would take. e.g. if 2,000 by 2,000 takes 1 second, then 20,000 by 20,000 should scale up to about 1,000 seconds.

This was the first time I've heard my fan rev up to full speed in a long time...