My simulation, written in C++, generates a large amount (roughly ~500) of text files for each set of parameters I try to simulate, with four columns of ~5k double values in each file. Furthermore, to keep track of the used parameters, I print all the used parameters in the name of the files, followed by the current file number. This makes it easy for tracking, but the more parameters I add, the longer and more complicated the file name.
Thus, I am looking for a way to make that more compact. One way I thought of would be using databases like HDF5, which would allow me to put everything into one file, while retaining information about the simulation parameters. But afterwards I would like to plot single files using tools like GnuPlot, requiring unpacking the data again, which I would like to avoid.
Another way I saw was using Sumatra (http://neuralensemble.org/sumatra/), but my program will run on a HPC-cluster, and I am not aware of a possibility to couple Sumatra with the job scripts necessary for slurm, and slurm itself.
Thus, are there other options I could use?

  • 2
    $\begingroup$ There are HDF5 wrappers for Python, and it's easy to plot data in python. $\endgroup$ – Wolfgang Bangerth Aug 20 '19 at 22:36

You should be able to store everything in one table and plot from that if your data is tidy. Check out this paper, it's only about 20 pages.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.