# Storing a large, sparse array for R and Python

I've been working in R but sometimes switching to python. I'd like a more inter-language portable way of storing a large array than a csv file. (The particular csv file I'm dealing with is about 10^6 rows by 10^3 columns, but only about 1% of the entries are non-zero.) Processing a large csv file everytime I start up R or Python takes far too long.

I've heard HDF5 is a great solution for this, but I have limited experience with it. Is HDF5 appropriate for storing non-hierarchical array data? What about for a single sparse array? I also am unsure which HDF5 R package to use.

If you are interested in a "standardized" format, you should have a look at the matrix market exchange formats. The "coordinate format" (which is good for sparse matrices) simply adds metadata to the format suggested by Aron in his answer and specifies how the data has to be formatted.

• Indeed. In Python you can even use scipy.io.mmwrite and scipy.io.mmread to handle this for you. I suspect that R has similar library functions. – MRocklin Aug 19 '12 at 17:22
• @MRocklin The R Matrix package has functions for reading and writing matrix market format (readMM and writeMM). – Zach Jan 30 '14 at 16:34
• Where can I find more information about the coordinate format? – user1170330 Jan 19 '16 at 22:37
• @user1170330 simply on the matrix market page cited above math.nist.gov/MatrixMarket/formats.html#coord – Stefano M Jan 19 '16 at 23:52

Assuming your sparse array is 2-dimensional, you can decompose it into three vectors of column (index), row (index), and value fairly easily with a single traversal of the matrix. You can then store these vectors in whatever file format you want, no need to switch to HDF5 for that reason alone.

I happened to have this little Python function kicking around:

def storeSparseProblem(sp_mat, rhs, filename):
# Convert the sparse matrix to lil
sp_mat = sp_mat.tocoo()

# Make a record array out of the above
rcv = recarray((3, len(sp_mat.row)), dtype=[("row","<i8"), ("col","<i8"), ("val","<f8")])

# Set the values
rcv["row"] = sp_mat.row
rcv["col"] = sp_mat.col
rcv["val"] = sp_mat.data

# Open the output h5
h5 = openFile(filename, "w")

# Save the matrix
h5.createTable("/", "M", rcv, title="The Matrix")