suggestion for managing simulation runs?

Question

This questions may be a bit off-topic in comp-sci. if it is needed please suggest where does it fit with.

The question is regarding on how to manage all the simulation runs efficiently.

let's say, for instance, a simulation requires fixing 2 parameters which has to be defined at a certain suggested value range.

To find a better outcome produced by a pair of the two parameters (by comparing the simulation outcome with experimental data, for example), one can do sensitivity analysis by defining three values for each parameter, then formulating 9 runs.

previously i use sed to change inputs for each run, and tag each run by writing the value and parameter name on the folder that stores inputs and outcomes of this run. but I found this is very inefficient once the number of parameters grows (e.g. access the name of the folders in the scripts for plotting).

then I decided to use simple numbers as folder names and store the detail by some other spreadsheets. this way is ok so far but requires some laborious work. also with the growth of the runs, it becomes common to make mistakes, such as conduct another run which has already been done a couple of days ago.

Do you have any good idea about managing these runs? I think it would be extremely important for someone doing Monte Carlo analysis?

Thanks in advance!

Answer 1

TLDR
Use Python to manage/modify your input and coral your output, and use HDF5 to organize/store your data. As complex as it might seem at first it'll still be simpler than SQL-anything.

Longer answer + Example
I personally use a combination of Python scripting and the HDF5 file format to deal with these kinds of situations. Python scripting can handle the text substitutions necessary for altering your runfiles (and can check for duplicate runs), and with some more scripting you can take the output data from your program and put it into an HDF5 file.

It's easiest to think of HDF5 as being more or less exactly like a normal file system (ie the set of directories and subdirectories on your computer), but one that scales easily to large data sets. Each directory/subdirectory can be tagged with metadata (in your case either just the parameters that you're varying, or the entire set of parameters). When it comes time to analyze your data you can search through it based on the metadata.

Here's a short example of how this would work based on some of my simulation data (already in HDF5 format) that looks like this:

mydata.hdf5
|___Run01(metadata: {size:13, maxSteps:1e7, maxTime:inf})
|___Run02(metadata: {size:10, maxSteps:1e6, maxTime:inf})
|___Run03(metadata: {size:13, maxSteps:1e7, maxTime:inf})
|___Run04(metadata: {size:9, maxSteps:1e7, maxTime:inf})

mydata.hdf5 is the HDF5 file, and each of Runxx is a subdirectory that holds the output data from a given simulation, and which is tagged with the associated metadata. A python script that searches through the runs and return a list of those with the desired metadata would look like this:

import sys
import h5py    #the python module that interfaces with HDF5

def GetRuns(hdfRoot, attributeValuePairs):
    return [subdir for subdir in hdfRoot.values() if not(attributeValuePairs.viewitems() - dict(subdir.attrs).viewitems())]

if __name__=="__main__":
    attributeValuePairs = dict(zip(sys.argv[2::2], sys.argv[3::2]))
    with h5py.File(sys.argv[1]) as hdfRoot:
        runs = GetRuns(hdfRoot, attributeValuePairs)

        #do something here with runs...

        print runs

So if I was at a command line in a directory containing mydata.hdf5 I could run the above script like so:

python myscript.py mydata.hdf5 maxSteps 1e7 size 13

which would tell the script to find any runs with metadata partially or wholly matching {'maxSteps':'1e7', 'size':'13'}. The script could then manipulate that data however you liked (in the "do something here" section), and then it would print a list that would look something like this:

["Run01", "Run03"]

One note though is that HDF5 is going to present a totally natural mapping for your data only if it is possible to represent your data as a set of n-dimensional arrays. It's pretty common for the output of simulations to be in some kind of array, so this probably won't be an issue.

Good starting points
Python: http://www.openbookproject.net/thinkcs/python/english2e/
HDF5: http://www.h5py.org/docs/

Answer 2

I think we'd need to know a little bit more about your workflow to make any serious recommendations.

I'd suggest treating your runs like a key-value store. Create a simple database for all of your metadata for each run, and then hash any relevant information from your run into a key that you assign to each output.

In the simplest situation, you would use a text file for your metadata store, and safely append lines of metadata about each run to your text file. You can then store your output runs however you like (a single directory, backups with a listing of the contents, etc...)

You can implement this strategy in any language you like, but this would be trivial in Python. You could also take advantage of some nice features as Python's ability to read and write JSON data or interact with SQL databases.

This approach implements a very simple light-weight database. There are heavier strategies that provide more safety guarantees, a new one you might be interested in is SciDB. Databases provide stronger guarantees about your data and help you scale your approach for larger datasets.