I am currently running a large amount parameter variation experiments. They are being run in Python 2.6+, using numpy. These experiments are going to take about 2 weeks to run.
Roughly I am varying 3 parameters (independent variables) over a range of values. I am fixing 6 further independent variables (for now) I am reporting on 4 dependent variables.
One of the parameters I am varying is being distributed across several processes (and computers).
For each of these parameters, I generate a separate
csv files with each row containing the values of all the variables (including independent, fixed and dependent).
Across all the variation expect to generate about 80,000 rows of data
Most of the time I am only looking at the value of one of the dependent variables, however I keep the others around, as they can explain what is going on when something unexpected happens.
In a earlier version of this experiment, varying across only 2 parameters (each though only 2 values)
I was copying pasting this
csv file into a spreadsheet program and doing a bunch of copy pasting to make a table of just the dependent variable I was interested in.
The doing some awkward things in MS-Excel to let me sort by formulas.
This was painful enough for the 6 experiment results sets I had.
By the time this run is finished I am going to have 2 orders of magnitude more results.
I was thinking once done, I could dump all the results from the
csv files into a database, and the query out the parts that are interesting.
Then take those results and put them into a spreadsheet for analysis.
Making graphs, finding scored relative to the control results etc
Am I thinking along the right lines? (Is this what people do?)
My database foo is fairly rusty these days, even when it was good I was using MS-Access. I was intending on using MS-Access for this as well.