# Storing a large, sparse array for R and Python

I've been working in R but sometimes switching to python. I'd like a more inter-language portable way of storing a large array than a csv file. (The particular csv file I'm dealing with is about 10^6 rows by 10^3 columns, but only about 1% of the entries are non-zero.) Processing a large csv file everytime I start up R or Python takes far too long.

I've heard HDF5 is a great solution for this, but I have limited experience with it. Is HDF5 appropriate for storing non-hierarchical array data? What about for a single sparse array? I also am unsure which HDF5 R package to use.

-
 Just a side note: for python there is sparse matrix support in docs.scipy.org/doc/scipy/reference/sparse.html For R you can check stackoverflow.com/q/1167448/1499402 and stackoverflow.com/a/10717741/1499402 – Stefano M Aug 14 '12 at 7:51 Another side note: h5py (code.google.com/p/h5py) is a very easy to use HDF5 library for Python. – AlexE Aug 14 '12 at 8:18

If you are interested in a "standardized" format, you should have a look at the matrix market exchange formats. The "coordinate format" (which is good for sparse matrices) simply adds metadata to the format suggested by Aron in his answer and specifies how the data has to be formatted.

-
 Indeed. In Python you can even use scipy.io.mmwrite and scipy.io.mmread to handle this for you. I suspect that R has similar library functions. – MRocklin Aug 19 '12 at 17:22

Assuming your sparse array is 2-dimensional, you can decompose it into three vectors of column (index), row (index), and value fairly easily with a single traversal of the matrix. You can then store these vectors in whatever file format you want, no need to switch to HDF5 for that reason alone.

-
And if you want to look for how this array is usually described, search the web for "compressed sparse row (CSR)". For example here: en.wikipedia.org/wiki/… – Wolfgang Bangerth Aug 14 '12 at 6:49
@WolfgangBangerth the format described by Aron is "coordinate list": en.wikipedia.org/wiki/Sparse_matrix#Coordinate_list_.28COO.29 – Stefano M Aug 14 '12 at 7:10

I happened to have this little Python function kicking around:

def storeSparseProblem(sp_mat, rhs, filename):
# Convert the sparse matrix to lil
sp_mat = sp_mat.tocoo()

# Make a record array out of the above
rcv = recarray((3, len(sp_mat.row)), dtype=[("row","<i8"), ("col","<i8"), ("val","<f8")])

# Set the values
rcv["row"] = sp_mat.row
rcv["col"] = sp_mat.col
rcv["val"] = sp_mat.data

# Open the output h5
h5 = openFile(filename, "w")

# Save the matrix
h5.createTable("/", "M", rcv, title="The Matrix")