I have matrices that are extremely easy to compute pointwise, but are too large to store. (they are not sparse) On the MATLAB site I was told MATLAB doesnt support computations with non-stored matrices. Is their any other software package out there that does?
$\begingroup$
$\endgroup$
4
-
$\begingroup$ I assume that with storing you are referring to RAM, but the matrix would stored on a HHD? Or do you have a function which can calculate the value of a given index of the matrix? $\endgroup$– fibonaticMar 11, 2015 at 21:28
-
2$\begingroup$ How big are the matrices? If the matrices are so big that they can't be stored explicitly, yet are dense, wouldn't even forming matrix-vector products be overly expensive? PETSc can work with non-stored matrices. $\endgroup$– KirillMar 11, 2015 at 21:55
-
$\begingroup$ the matrix can be calculated by indices $\endgroup$– BananachMar 11, 2015 at 21:56
-
$\begingroup$ Where does the linear system of equations come from? In many cases large dense systems of equations have structure that can be exploited by specialized methods, but these methods depend on the particular structure of the system of equations, so you'd need to start by telling us where the equations came from. $\endgroup$– Brian BorchersMar 12, 2015 at 1:16
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
1
If the individual entries of the matrices involved can be calculated on the fly relatively cheaply, as you say, then you could turn to matrix-free methods (also known as black box solvers). Krylov-subspace methods such as GMRES belong to that class of solvers and would be useful for you because they only require that the user calculates matrix-vector products. You can do these matrix-vector products by computing the relevant entries of your matrix on the fly, without storing the whole matrix in memory.
The reverse communication routines in the Fortran version of the code available in Templates for the Solution of Linear Systems would match your requirements.
answered Mar 12, 2015 at 0:18
-
5$\begingroup$ MATLAB can do matrix-free GMRES if you give it a function handle that computes the action of the matrix on a vector. $\endgroup$ Mar 12, 2015 at 1:41
$\begingroup$
$\endgroup$
2
If the matrices are too large to fit into RAM but reasonably could fit on your hard disk, then you should look at what are called "Out of Core" solvers. These solvers partition the large matrix into blocks that are small enough to be handled one at a time in memory. There are many available codes for both dense and sparse problems, but getting good performance out of an out of core solver often requires tuning the code to the particular I/O performance of the computer that you're using.
answered Mar 11, 2015 at 21:57
-
$\begingroup$ thanks, I'll have a look at it. however, as specified in the comments above, the entries of the matrix can be calculated by entries. therefore, i assume storage on HDD will be a huge and unnecessary waste of time $\endgroup$– BananachMar 11, 2015 at 22:01
-
1$\begingroup$ I should have clarified that OOC solvers use direct factorization techniques rather than iterative methods. In the factorization of the matrix you will typically produce intermediate results that are of the same size as the original matrix and these will have to be stored on disk- recreating the elements of the array as needed isn't really an option. If you're using an iterative scheme, then you can probably compute matrix-vector multiplications without ever explicitly computing elements of the matrix. $\endgroup$ Mar 12, 2015 at 1:13