Somewhat related, but I think the question is distinct enough to justify a separate question.
As a bit of background, I come from a observational/statistical Epidemiology background, working with data that's been collected already, so even our large datasets tend to be in discrete, inseparable chunks. As such, I've never actually learned how to handle large amounts of data coming out of a simulation.
My problem is as follows:
I'm working on a compartmental model of an infection system, involving a system of ~10 differential equations and ~40 parameters shared among them. Most of these parameters aren't constants, but rather statistical estimates, and as such could be drawn from a distribution. That's one of the things I'd like to do - see how much the system varies purely as the result of uncertainty in the parameter estimates.
Which involves running the numerical solution to the model sampling from each distribution many many times to cover the parameter space. If it was just a stochastic simulation, I might be fine with outputting one huge, or thousands of small data files that I could churn through with a script. My current problem is how to manage the output data given I need to know what parameter values were drawn.
Right now, my very quick and dirty method is to output the parameter value alongside the numerical results, meaning that if I ran the system for 100 steps, I end up with a 100 row, 50 column data set - but where 40 of those columns are the same number repeated over and over again. That seems hugely wasteful, and is resulting in really large files.
Surely there's a better way to do this? Most of this is, currently, being implemented in Python.