Storing a large, sparse array for R and Python

Question

I've been working in R but sometimes switching to python. I'd like a more inter-language portable way of storing a large array than a csv file. (The particular csv file I'm dealing with is about 10^6 rows by 10^3 columns, but only about 1% of the entries are non-zero.) Processing a large csv file everytime I start up R or Python takes far too long.

I've heard HDF5 is a great solution for this, but I have limited experience with it. Is HDF5 appropriate for storing non-hierarchical array data? What about for a single sparse array? I also am unsure which HDF5 R package to use.

Answer 1

If you are interested in a "standardized" format, you should have a look at the matrix market exchange formats. The "coordinate format" (which is good for sparse matrices) simply adds metadata to the format suggested by Aron in his answer and specifies how the data has to be formatted.

Answer 2

Assuming your sparse array is 2-dimensional, you can decompose it into three vectors of column (index), row (index), and value fairly easily with a single traversal of the matrix. You can then store these vectors in whatever file format you want, no need to switch to HDF5 for that reason alone.

Answer 3

I happened to have this little Python function kicking around:

def storeSparseProblem(sp_mat, rhs, filename):
    # Convert the sparse matrix to lil
    sp_mat = sp_mat.tocoo()

    # Make a record array out of the above
    rcv = recarray((3, len(sp_mat.row)), dtype=[("row","<i8"), ("col","<i8"), ("val","<f8")])

    # Set the values
    rcv["row"] = sp_mat.row
    rcv["col"] = sp_mat.col
    rcv["val"] = sp_mat.data

    # Open the output h5
    h5 = openFile(filename, "w")

    # Save the matrix
    h5.createTable("/", "M", rcv, title="The Matrix")

    # Add the rhs
    ary = h5.createArray("/", "rhs", rhs, title="the right hand side")

    # Close it out
    h5.close()

You could make this faster by turning on tables's magic blosc compression, and switching from a table to a carray, but this should at least get you started. The load function is really simple, but I can't seem to locate it. If someone has interest, I will code it up and post it.