I've been running various simulations with C++, and doing so has often involved saving lots of data to file (real/complex matrices, arrays, etc) and then reading them into other programs later. However, I have been unable to figure out a consistent and simple way of doing so. My solutions often involve several dozen lines of difficult-to-read code for each different kind of data structure that I want to save/read from a file. Are there any known libraries that can do this, or at least aid in the process? If there are any sort of matrix libraries that allow automatic saving/reading of matrices/vectors to file, that would also be helpful (Eigen does not have this option, as far as I can tell).
$\begingroup$
$\endgroup$
3
-
4$\begingroup$ Will HDF5 work? $\endgroup$– user14717Feb 18, 2019 at 19:18
-
2$\begingroup$ What is your goal? To read the data you write from other programs? If all you want is to read the data into your own program again, there is little wrong with just doing a memory dump of your data structure in a format of your choice. But it's a different matter if you want to interact with other programs. $\endgroup$– Wolfgang BangerthFeb 19, 2019 at 7:50
-
2$\begingroup$ @WolfgangBangerth makes a good point or two. But don't forget that the definition of other programs includes any program you wrote more than about 1 week ago and for which you have already started to forget (OK, maybe 2 weeks, I'm getting on) the precise details of the file structure and contents. Files which contain their own metadata are a tremendous boon. $\endgroup$– High Performance MarkFeb 20, 2019 at 11:39
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
Numpy has a file format that is pretty simple, which makes it perfectly compatible with basically every other high level language. (https://www.numpy.org/devdocs/reference/generated/numpy.lib.format.html) It looks like the format is much lighter than boost or hdf5. The docs say it should be easy enough to write a parser yourself, if necessary, which I would tend to believe. Cursory googling also suggests there are a number of existing lightweight libraries in C++ for .npy files (such as https://github.com/rogersce/cnpy). As as added benefit, you'd always be able to load your files in python and examine them by hand in a nice python interpreter like ipython, which can be a little tricky to achieve in C++ otherwise.
$\begingroup$
$\endgroup$
1
You can use boost's serialization facilities, which can be straightforwardly extended to support custom data structures. If you use Eigen you can adapt something like this in order to suit your needs.
-
$\begingroup$ I don't know why I didn't think of this when writing my comment above. Yes, use boost serialization. It's a great library. $\endgroup$ Feb 21, 2019 at 4:12
$\begingroup$
$\endgroup$
Cereal is a straight-forward and easy-to-use serialization library for C++ data structures. It knows all of the STL data structures and adapting custom classes for use with it is easy.
Example code:
#include <cereal/types/unordered_map.hpp>
#include <cereal/types/memory.hpp>
#include <cereal/archives/binary.hpp>
#include <fstream>
struct MyRecord
{
uint8_t x, y;
float z;
template <class Archive>
void serialize( Archive & ar )
{
ar( x, y, z );
}
};
struct SomeData
{
int32_t id;
std::shared_ptr<std::unordered_map<uint32_t, MyRecord>> data;
template <class Archive>
void save( Archive & ar ) const
{
ar( data );
}
template <class Archive>
void load( Archive & ar )
{
static int32_t idGen = 0;
id = idGen++;
ar( data );
}
};
int main()
{
std::ofstream os("out.cereal", std::ios::binary);
cereal::BinaryOutputArchive archive( os );
SomeData myData;
archive( myData );
return 0;
}
