I've been working in R but sometimes switching to python. I'd like a more inter-language portable way of storing a large array than a csv file. (The particular csv file I'm dealing with is about 10^6 rows by 10^3 columns, but only about 1% of the entries are non-zero.) Processing a large csv file everytime I start up R or Python takes far too long.
I've heard HDF5 is a great solution for this, but I have limited experience with it. Is HDF5 appropriate for storing non-hierarchical array data? What about for a single sparse array? I also am unsure which HDF5 R package to use.