Best way to track parameters and results, in your experience (i.e. how do you do it)?

Question

Computational Science has many more parameters than my previous programming, and I'm finding it difficult to keep track. What's the best way to track them?

1. editing source, commenting-out. Good for trying things quickly, but after some exploration, I've forgotten what the earlier ones were. Much harder when a setup requires changes to more than one part of the code (e.g. changing viscosity and initial conditions (IC)).

2. in a class For each experiment, write a class with a method for each part of the set-up, then call those methods from the appropriate parts of the main code (e.g. specific IC, and specific boundary conditions (BC), and a combination of values for $\nu$ and $\Delta x$, and excluding/switching off some aspect, like advection, and maybe also changing visualization parameters, like scaling - though that really should be separate). Having one class for all these separate changes keeps them together in one place. (To run an experiment, instantiate the class for it, and plug it into the main code.)

   • code: A further benefit of a class is that actual code is easily included e.g. It's common for IC to be defined as a function or method, not data points, because the former is more intelligible and modifiable. Sometimes, the experiment is to change mathematical expressions/discretizations (i.e. again, executable code).

3. file format Store the parameters in a file, and read it in and configure the appropriate parts of the main code. This doesn't work so well for choosing different mathematical expressions, unless you're using a dynamic language (e.g. python). Could use a proper database (e.g. SQL) instead of files.

Though these don't include the results, such as data, plots, images. The above are my programmer-centric thoughts, but it occurs to me that Computational Science may have a more "science" approach, i.e. methodically recording experimental inputs and results. It could be done by making notes on paper, or in some form easier for publication.

Perhaps also, standard software packages include means for easily recording inputs and results? e.g. automatically journaled, so you can easily go back, if and when you need to write-up or revisit a particular experiment?

I'm interested both in what is considered the best way... but also, what people you actually do, which might be better than the so-called "best" (which might be too rigid, adding complexity overhead).

EDIT We have limited attention, and code, paramaters, experiment and hypotheses can be complex enough, without unnecessary management overhead - especially if you're still learning (as I am). So, quick-and-dirty sometimes can be better trade-off, in practice, taking into account the true context and needs, than a "proper" heavy-weight industrial/institutional/enterprise grade solution.

Answer 1

Since you've asked about experiences, I've seen all three of the methods you suggest used. However best practise is that inputs to the code, and the code framework itself are kept separate. This means that only parameters you don't expect to ever need to change should be hard coded into your source. For example, if you were modelling the flight of cannonballs, you might hard code the value for gravitational acceleration, but not the radius of the cannonball, which forms part of the metadata for the individual simulation. That way the code base in your version control system doesn't get polluted with input data, and remains sane for someone else (or you six months later) to understand.

In terms of classes, versus configuration files for the simulation input data, that really comes down to the language being used and to whether you're developing an executable or a framework. Again, best practise is to use a self-documenting system like xml or JSON for configuration files (no need to roll your own implementation, there are prebuilt parsing libraries out there). Ideally the file should be human readable, either directly or via a helper utility. For the class based method, the attribute names should again be human understandable, although there are still people out there who are stuck in the FORTRAN 66 structure of six-letter variable names like "GRNEW4".

Outputting should again be in a commonly used, self-describing file format, and ideally should include enough metadata to be able to reproduce the original input data which generated it if necessary. Since input files tend to be small compared to the output data, one option is to simply prepend the input file/source at the start of the output data.

The method of organisation for the final output is probably the point of most debate. While version control systems work pretty well for input files (in a separate repository from the source code remember), they don't always play that well with binary data. This leads to a lot of disk based organisation (use human understandable file names!), but a useful possibility is to take advantage of one of the various generic reference managers like JabRef, which are readily available and provide searchable tracking of the simulation metadata.

Answer 2

I don't know what you can do to keep track of parameters and results, though @origimbo correctly noted that repeatability kinda makes results useless to store.

The first point you make is obviously the versioning problem, so the solution is trivial: create a repository on Github or Bitbucket or Mercurial. These services are very effective and they let you reverse a given commit if you messed up your code and you need to restore it. They are supported as plugin within most IDEs.

Plus: redundancy on cloud! Making backup copies will be a bit less urgent than it usually is.

Edit: implement a logger into your programs.

Answer 3

Manage parameters with switch statements: each case is a self-contained set of parameters. This doesn't address different expressions/codes etc, but for Barba's 12 steps, it's enough so far. EDIT and arbitrary boolean switches for different parts of the code (e.g. different boundary conditions; switching off a stage such as advection). Messy but flexible.

Add parameters to output plots: so the input and interpretable results are documented together, in an image file.

Automatically timestamping the plot images, and outputting to a tmp directory, so I can go back to them if need be, (and copy and rename any I want to keep). This is the journalling idea; and, I guess is a form of "logging".

I'm using gnuplot for plotting, and send in the parameters and filename as "call" arguments (this line is auto-generated from my simulation code):

gnuplot -c myscript filename stringofparameternamesandvalues

These argumemts are available as ARG1 and ARG2 in the script. In the script, I set the output file and title to them. One can also put them in a label, and position it (and use \n for multiline), but I'm concerned it will sometimes overwrite part of the plot - using the title prevents that.

Thought: a parameter object. Using reflection, one can automatically serialize to JSON (or XML), or output to a logger (as @GoGoLander suggested). An advantage is if you add a new parameter to the object, you don't have to remember to add it the output code (the serializer automatically output all the object's fields). Though this doesn't capture expressions/code. EDIT a parameter object is verbose because must add prefix (e.g. params.dx). Instead, use fields on the code's own instance object as parameters, and can use them directly (e.g. dx). Here's a way to auto-output them in java (assuming no other fields):

import java.lang.reflect.Field;
...
    public String getParams() {
      String a = "";
      try {
        for (Field f : this.getClass().getDeclaredFields()) {
          f.setAccessible(true);  // so needn't make fields public
          a += f.getName() + "="+ f.get(this) +", ";
        }
      } catch (Exception e) { e.printStackTrace(); }
      return a;
    }

I think "2. in a class" is the right approach for compiled languages, because it can store expressions and code, and keeps experimental config separate from the main code.