5
$\begingroup$

I've been working in R but sometimes switching to python. I'd like a more inter-language portable way of storing a large array than a csv file. (The particular csv file I'm dealing with is about 10^6 rows by 10^3 columns, but only about 1% of the entries are non-zero.) Processing a large csv file everytime I start up R or Python takes far too long.

I've heard HDF5 is a great solution for this, but I have limited experience with it. Is HDF5 appropriate for storing non-hierarchical array data? What about for a single sparse array? I also am unsure which HDF5 R package to use.

$\endgroup$
4
$\begingroup$

If you are interested in a "standardized" format, you should have a look at the matrix market exchange formats. The "coordinate format" (which is good for sparse matrices) simply adds metadata to the format suggested by Aron in his answer and specifies how the data has to be formatted.

$\endgroup$
  • $\begingroup$ Indeed. In Python you can even use scipy.io.mmwrite and scipy.io.mmread to handle this for you. I suspect that R has similar library functions. $\endgroup$ – MRocklin Aug 19 '12 at 17:22
  • $\begingroup$ @MRocklin The R Matrix package has functions for reading and writing matrix market format (readMM and writeMM). $\endgroup$ – Zach Jan 30 '14 at 16:34
  • $\begingroup$ Where can I find more information about the coordinate format? $\endgroup$ – user1170330 Jan 19 '16 at 22:37
  • $\begingroup$ @user1170330 simply on the matrix market page cited above math.nist.gov/MatrixMarket/formats.html#coord $\endgroup$ – Stefano M Jan 19 '16 at 23:52
2
$\begingroup$

Assuming your sparse array is 2-dimensional, you can decompose it into three vectors of column (index), row (index), and value fairly easily with a single traversal of the matrix. You can then store these vectors in whatever file format you want, no need to switch to HDF5 for that reason alone.

$\endgroup$
2
$\begingroup$

I happened to have this little Python function kicking around:

def storeSparseProblem(sp_mat, rhs, filename):
    # Convert the sparse matrix to lil
    sp_mat = sp_mat.tocoo()

    # Make a record array out of the above
    rcv = recarray((3, len(sp_mat.row)), dtype=[("row","<i8"), ("col","<i8"), ("val","<f8")])

    # Set the values
    rcv["row"] = sp_mat.row
    rcv["col"] = sp_mat.col
    rcv["val"] = sp_mat.data

    # Open the output h5
    h5 = openFile(filename, "w")

    # Save the matrix
    h5.createTable("/", "M", rcv, title="The Matrix")

    # Add the rhs
    ary = h5.createArray("/", "rhs", rhs, title="the right hand side")

    # Close it out
    h5.close()

You could make this faster by turning on tables's magic blosc compression, and switching from a table to a carray, but this should at least get you started. The load function is really simple, but I can't seem to locate it. If someone has interest, I will code it up and post it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.