My simulation, written in C++, generates a large amount (roughly ~500) of text files for each set of parameters I try to simulate, with four columns of ~5k double values in each file. Furthermore, to keep track of the used parameters, I print all the used parameters in the name of the files, followed by the current file number. This makes it easy for tracking, but the more parameters I add, the longer and more complicated the file name.
Thus, I am looking for a way to make that more compact. One way I thought of would be using databases like HDF5, which would allow me to put everything into one file, while retaining information about the simulation parameters. But afterwards I would like to plot single files using tools like GnuPlot, requiring unpacking the data again, which I would like to avoid.
Another way I saw was using Sumatra (http://neuralensemble.org/sumatra/), but my program will run on a HPC-cluster, and I am not aware of a possibility to couple Sumatra with the job scripts necessary for slurm, and slurm itself.
Thus, are there other options I could use?
$\begingroup$
$\endgroup$
1
-
2$\begingroup$ There are HDF5 wrappers for Python, and it's easy to plot data in python. $\endgroup$ – Wolfgang Bangerth Aug 20 '19 at 22:36
Add a comment
|
$\begingroup$
$\endgroup$
You should be able to store everything in one table and plot from that if your data is tidy. Check out this paper, it's only about 20 pages.
