5
$\begingroup$

I've been working in R but sometimes switching to python. I'd like a more inter-language portable way of storing a large array than a csv file. (The particular csv file I'm dealing with is about 10^6 rows by 10^3 columns, but only about 1% of the entries are non-zero.) Processing a large csv file everytime I start up R or Python takes far too long.

I've heard HDF5 is a great solution for this, but I have limited experience with it. Is HDF5 appropriate for storing non-hierarchical array data? What about for a single sparse array? I also am unsure which HDF5 R package to use.

$\endgroup$
2

3 Answers 3

4
$\begingroup$

If you are interested in a "standardized" format, you should have a look at the matrix market exchange formats. The "coordinate format" (which is good for sparse matrices) simply adds metadata to the format suggested by Aron in his answer and specifies how the data has to be formatted.

$\endgroup$
4
  • $\begingroup$ Indeed. In Python you can even use scipy.io.mmwrite and scipy.io.mmread to handle this for you. I suspect that R has similar library functions. $\endgroup$
    – MRocklin
    Aug 19, 2012 at 17:22
  • $\begingroup$ @MRocklin The R Matrix package has functions for reading and writing matrix market format (readMM and writeMM). $\endgroup$
    – Zach
    Jan 30, 2014 at 16:34
  • $\begingroup$ Where can I find more information about the coordinate format? $\endgroup$
    – user18957
    Jan 19, 2016 at 22:37
  • $\begingroup$ @user1170330 simply on the matrix market page cited above math.nist.gov/MatrixMarket/formats.html#coord $\endgroup$
    – Stefano M
    Jan 19, 2016 at 23:52
2
$\begingroup$

Assuming your sparse array is 2-dimensional, you can decompose it into three vectors of column (index), row (index), and value fairly easily with a single traversal of the matrix. You can then store these vectors in whatever file format you want, no need to switch to HDF5 for that reason alone.

$\endgroup$
2
2
$\begingroup$

I happened to have this little Python function kicking around:

def storeSparseProblem(sp_mat, rhs, filename):
    # Convert the sparse matrix to lil
    sp_mat = sp_mat.tocoo()

    # Make a record array out of the above
    rcv = recarray((3, len(sp_mat.row)), dtype=[("row","<i8"), ("col","<i8"), ("val","<f8")])

    # Set the values
    rcv["row"] = sp_mat.row
    rcv["col"] = sp_mat.col
    rcv["val"] = sp_mat.data

    # Open the output h5
    h5 = openFile(filename, "w")

    # Save the matrix
    h5.createTable("/", "M", rcv, title="The Matrix")

    # Add the rhs
    ary = h5.createArray("/", "rhs", rhs, title="the right hand side")

    # Close it out
    h5.close()

You could make this faster by turning on tables's magic blosc compression, and switching from a table to a carray, but this should at least get you started. The load function is really simple, but I can't seem to locate it. If someone has interest, I will code it up and post it.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.