What is a good way to run parameter studies in C++

Question

The problem

I'm currently working on a Finite Element Navier Stokes simulation and I would like to investigate the effects of a variety of parameters. Some parameters are specified in an input file or via a command line options; other parameters are provided as flags in a Makefile so my code has to be recompiled whenever I change those options. I would be interested to get some advice about a good way to systematically explore the parameter space.

• Are there useful C++/Python libraries/frameworks that can help with this sort of thing? For example discovering boost.Program_options was a big help since it's possible to overload input file options with command line arguments. I have also seen some people use a job file describing each case quite effectively and a collegue suggested that writing parameters into vtu files as comment blocks could work too.
• Perhaps it isn't worth investing much time in this at all? Is it just a distraction and a time-drain and it's best to just muscle through the testing process brute force and ad hoc?

Some thoughts

I am currently doing things mostly by hand and I have encountered the following problems:

• Naming test cases. I tried storing results in folders named with the run parameters separated with underscores e.g. Re100_dt02_BDF1.... These quickly become long or difficult to read/cryptic if they are abbreviated too much . Also, real number parameters include a . which is awkward/ugly.
• Logging run data. Sometimes I would like to see the results written to the terminal and also saved to a text file. This answer from StackOverflow for instance is somewhat helpful but the solutions seem to be a bit intrusive.
• Plotting data according to parameter. It takes quite some time collect relevant data from a variety of log files into a single file which I can then plot, with a better system perhaps this would become easier.
• Recording comments on the data. After examining results I write some comments in a text file but keeping this is sync with the results folders is sometimes difficult.

Answer 1

Just some comments on two of your points:

• Logging run data: Your best bet is probably piping output through the tee command, which should be available in most shells.

• Plotting data according to parameter: I guess it's a matter of taste, but when I have to do complex data aggregation, I store the results in plain text, read them into Matlab as matrices, and do all the computations, plotting and even LaTeX output from there. Obviously, whatever programming/scripting language you're most familiar with will give you best results.

Answer 2

If you want to write something general-purpose, you can do it either with shell scripts if it is something very simple, as Pedro suggests, or aggregate in a higher-level mathematical programming language such as Python or MATLAB. I agree that plain text files are useful for smaller amounts of data, but you should probably switch to binary data for anything larger than a few megabytes.

On the other hand, if you are just doing parameter estimation, I would recommend using a piece of software specifically suited for this. Several researchers at my University have had good luck with DAKOTA, an Uncertainty Quantification toolbox out of Sandia National Laboratories (available under a GNU Lesser General Public License).

Here's an excerpt from the Sandia page describing DAKOTA:

We provide a variety of methods to allow a user to run a collection of computer simulations to assess the sensitivity of model outputs with respect to model inputs. Common categories include parameter studies, sampling methods and design of experiments. In parameter studies one steps some input parameters through a range while keeping other input parameters fixed and evaluates how the output varies. In sampling methods, one generates samples from an input space distribution and calculates the output response at the input values. Specific sampling methods available within DAKOTA include Monte Carlo, Latin Hypercube, and (coming soon) quasi-Monte Carlo. In design of experiments the output is evaluated at a set of input "design" points chosen to sample the space in a representative way. Specific design of experiment methods available within DAKOTA include Box-Behnken, Central Composite, and Factorial designs. Sensitivity metrics are a mathematical way of expressing the dependence of outputs on inputs. A variety of sensitivity metrics are available within Dakota, such as simple and partial correlation coefficients, and rank correlations. Our current research focuses on methods to generate sensitivity metrics with a minimal number of runs, and on optimal estimation of parameters in computer models using Bayesian analysis techniques.

Answer 3

For my doctoral work, I am running into similar issues as you are. Since it is not my code that I am using, though, I do not have quite the same flexibility as you do. That said, I do have a few suggestions.

As Pedro suggested, there is the tee command. But, if it is not available, or you would like something built into your software itself, I would suggest looking at the boost::iostreams library. It provides mechanisms for defining input sources and output sinks which the standard library does not do. In particular, there is the tee_device which allows you to connect two output sinks to your stream, and other streams can act as sinks. This would allow you to make the simultaneous output to stdout and a log-file config dependent.

I agree that boost::program_options can be very helpful in configuring your software. However, it has a couple flaws which may impact how you do things, though. First, if you need a hierarchical configuration,$^1$ then ini files are a painful way of accomplishing it. Second, and more significantly, boost::program_options has no output capabilities, so you cannot save your state as a configuration file for later inspection or resuming a stopped code. As an alternative, I would suggest using boost::property_tree which supports hierarchical configuration files and saving the trees for later re-use. This has the added benefit that if you need to checkpoint your code, you can save out its current state as input when you restart.

For gathering the data from the different calculations, I loop over all the data files I would like to include in a set, I then use awk to produce a single line in the file, and pipe all the results into my output. This can take a couple of minutes, but unfortunately, I do not have a better method.

As to processing/commenting on your data, I can not stress the usefulness of the Mathematica notebook format enough. It allows me to organize my observations, speculations, and visualizations all in one place. My notebooks regularly top 100 MB, though. For good measure, Mathematica performs just as well as Matlab on matrix tasks. Additionally, it can be used to take notes with full mathematical formatting in real time.

I wish I had a better solution to the naming problem, and it is rather pernicious. It may be worthwhile considering outputting some of your data into a database because of this. However, if you do not wish to do that, either consider using the extended attributes in XFS to capture more complete information about your simulation, and store your config file in with the data that it was used to generate.

_{1. As an example where hierarchical configuration files is needed, a friend of mine was examining the effects of different tip geometries in AFM and each geometry had a different set of parameters. Additionally, alongside this, he was testing several calculation schemes so he could compare them to experiment, and they had vastly different parameters.}

Answer 4

I get to know PyTables when installing PETSC. And I guess the table (or database) method is well-suited for exploring parameter space, though I have not tried yet. We can record every run with specific parameters and then we can consult any aggregations satisfying some conditions, say, we can fix dt, BDF1 and look up all relevant records to study the variation due to the other parameters.

I would like to hear from people who are actually using the table (or database) method for exploring parameter space. I will appreciate for detailed expamples.

Answer 5

Exploring parameter space like you are trying to do can very quickly become unwieldy. There are so many different ways of doing this that there is no one real solution.

Usually when you reach this limit in your work, you might want to investigate hierarchical data formats HDF5. HDF5 allows you to store complex output of your simulation in a well defined file format. Advantages are that your data is stored in a single well defined file format. You can add multiple simulation runs, identified by different parameters, to your file, and manipulate them afterwards. The data can be compressed and is fairly easy to extract out using a variety of tools. There are easy to apis for c/c++/python etc and plenty of command line tools to manipulate the files. A disadvantage is that writing to hdf5 is not quite as simple as writing to the console. There are many example programs at HDF5 examples.

Answer 6

You want to keep an indexed table of variable values. The index corresponds to a folder where you keep each simulation input and output. So it's just an index and you don't have to worry about naming convention or folder hierarchies because you'll look up what parameters values correspond to each folder.

So now you can use this table to organize your post-processing, plotting (analysis), logging, and commenting. The table is central to the workflow.

This is the basic idea, and I'm describing what you might want to do only conceptually. In my initial response, I suggested looking into the framework that I've developed. More recently I've discovered Sumatra. It's much more developed than my individually-developed, struggling grad student, and new to python effort but I think it tries to do too much. It's focused on provenance info while my framework focuses on workflow efficiency. There is also jobman, sacred, and lencet.

Whatever you choose to do, I strongly recommend python to tackle these types of tasks since you can manage your whole workflow with python. Just as a little story, I watched my colleagues work with DAKOTA, bash, GNUplot, file naming conventions, sed/awk octave...etc. to do their computational work. Each of these tools are fine on their own but the power of python as an integrating glue language really shines when you use python for managing your work along with the python scientific stack. I literally had zero issues managing my computational work after I developed my framework.

/my initial response follows/

I believe I have solved this problem using python. I have thought of all these issues.

Check out my repo http://msdresearch.blogspot.com/2012/01/parameter-study-management-with-python.html

As of now though, I'm working on better documenting my framework. (it's more involved than filling in a readme!)

-Majid alDosari

Answer 7

I tend to agree in the following implementation, which I developed on the course of my investigation work, as can be found here, here and here.

To pass variables to the program and be able to change then, I use the paradigm of using a bash script where I define

export aValue=10
export bValue=2
export idName=test

and then use in C/C++

char *env_aValue = getenv("aValue");
char *env_bValue = getenv("bValue");
char *env_idName = getenv("idName");

aValue = atoi(env_aValue)
...

The big advantages of this is that:

• it can be accessed in a global scope,
• it is portable to sun grid engine (clusters),
• can be easily changed on the bash script,
• it is platform independent,
• the number of parameters can be very large (potentially infinite)

Besides, I always pass an idName, on which every file written by that executable will have a initial identification of it (can be followed by other parameters if you want), and they also receive a export directory=idName, which is created on the bash script, and all the files of that executable are saved on it. This way the results are organized by directories (optional).

Answer 8

You can check out sfepy which is a finite element program almost entirely coded in python. It also has sample Navier Stokes problem. The operating procedure of sfepy is very easy.

Answer 9

Have you thought about using a MySQL database? I've never done it, but I could imagine, that you can query this system very good! Perhaps other systems like MongoDB are better. So, this is just an idea.