For example I have 2 matrices that can't fit in RAM. I need algorithm or library which can handle this.Preferably Matlab or Python.

I think it can be some block matrix multiplication? Also I think there is an analogy hard drive<->ram, gpu ram<->cpu ram, cpu ram<->cpu cache, so we can take some cpu cache optimized techniques?

It seems in python I can use numpy.memmap but I don't understand memory consumption of this approach and maybe it isn't optimal solution at all.


I think you should have a look at PyTables. Especially the tutorial given at PyData 2012. PyTables combines hierarchical datasets with a computational engine. It uses the Blosc compresser to avoid I/O bottlenecks and an optimized evaluator for expressions tables.Expr (based on Numexpr).

  • 1
    $\begingroup$ Can you provide any example of matrix multiplication using pytables? $\endgroup$ – mrgloom Oct 21 '13 at 13:19
  • $\begingroup$ PyTables does not appear to provide this, nor is there package that pulls data in reasonable chunks into SciPy matrices. $\endgroup$ – Brian Dolan Jul 18 '17 at 21:22

More info about your matrix type would help. The easiest way to work with large matrices like this is to distribute them across multiple machines and use ScaLAPACK to do the operations in parallel. It will be faster too. If you need to do it on one machine, the out-of-core techniques you were alluding to will work. pytables mentioned by mrgloom supports this. Generally you want to take advantage of the nature of your matrix.

  • $\begingroup$ Can you provide any example of matrix multiplication using pytables? $\endgroup$ – mrgloom Oct 21 '13 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.