TLDR
Use Python to manage/modify your input and coral your output, and use HDF5 to organize/store your data. As complex as it might seem at first it'll still be simpler than SQL-anything.
Longer answer + Example
I personally use a combination of Python scripting and the HDF5 file format to deal with these kinds of situations. Python scripting can handle the text substitutions necessary for altering your runfiles (and can check for duplicate runs), and with some more scripting you can take the output data from your program and put it into an HDF5 file.
It's easiest to think of HDF5 as being more or less exactly like a normal file system (ie the set of directories and subdirectories on your computer), but one that scales easily to large data sets. Each directory/subdirectory can be tagged with metadata (in your case either just the parameters that you're varying, or the entire set of parameters). When it comes time to analyze your data you can search through it based on the metadata.
Here's a short example of how this would work based on some of my simulation data (already in HDF5 format) that looks like this:
mydata.hdf5
|___Run01(metadata: {size:13, maxSteps:1e7, maxTime:inf})
|___Run02(metadata: {size:10, maxSteps:1e6, maxTime:inf})
|___Run03(metadata: {size:13, maxSteps:1e7, maxTime:inf})
|___Run04(metadata: {size:9, maxSteps:1e7, maxTime:inf})
mydata.hdf5
is the HDF5 file, and each of Runxx is a subdirectory that holds the output data from a given simulation, and which is tagged with the associated metadata. A python script that searches through the runs and return a list of those with the desired metadata would look like this:
import sys
import h5py #the python module that interfaces with HDF5
def GetRuns(hdfRoot, attributeValuePairs):
return [subdir for subdir in hdfRoot.values() if not(attributeValuePairs.viewitems() - dict(subdir.attrs).viewitems())]
if __name__=="__main__":
attributeValuePairs = dict(zip(sys.argv[2::2], sys.argv[3::2]))
with h5py.File(sys.argv[1]) as hdfRoot:
runs = GetRuns(hdfRoot, attributeValuePairs)
#do something here with runs...
print runs
So if I was at a command line in a directory containing mydata.hdf5
I could run the above script like so:
python myscript.py mydata.hdf5 maxSteps 1e7 size 13
which would tell the script to find any runs with metadata partially or wholly matching {'maxSteps':'1e7', 'size':'13'}
. The script could then manipulate that data however you liked (in the "do something here" section), and then it would print a list that would look something like this:
["Run01", "Run03"]
One note though is that HDF5 is going to present a totally natural mapping for your data only if it is possible to represent your data as a set of n-dimensional arrays. It's pretty common for the output of simulations to be in some kind of array, so this probably won't be an issue.
Good starting points
Python: http://www.openbookproject.net/thinkcs/python/english2e/
HDF5: http://www.h5py.org/docs/