# Best way to track parameters and results, in your experience (i.e. how do you do it)?

Computational Science has many more parameters than my previous programming, and I'm finding it difficult to keep track. What's the best way to track them?

1. editing source, commenting-out. Good for trying things quickly, but after some exploration, I've forgotten what the earlier ones were. Much harder when a setup requires changes to more than one part of the code (e.g. changing viscosity and initial conditions (IC)).

2. in a class For each experiment, write a class with a method for each part of the set-up, then call those methods from the appropriate parts of the main code (e.g. specific IC, and specific boundary conditions (BC), and a combination of values for $\nu$ and $\Delta x$, and excluding/switching off some aspect, like advection, and maybe also changing visualization parameters, like scaling - though that really should be separate). Having one class for all these separate changes keeps them together in one place. (To run an experiment, instantiate the class for it, and plug it into the main code.)

• code: A further benefit of a class is that actual code is easily included e.g. It's common for IC to be defined as a function or method, not data points, because the former is more intelligible and modifiable. Sometimes, the experiment is to change mathematical expressions/discretizations (i.e. again, executable code).
3. file format Store the parameters in a file, and read it in and configure the appropriate parts of the main code. This doesn't work so well for choosing different mathematical expressions, unless you're using a dynamic language (e.g. python). Could use a proper database (e.g. SQL) instead of files.

Though these don't include the results, such as data, plots, images. The above are my programmer-centric thoughts, but it occurs to me that Computational Science may have a more "science" approach, i.e. methodically recording experimental inputs and results. It could be done by making notes on paper, or in some form easier for publication.

Perhaps also, standard software packages include means for easily recording inputs and results? e.g. automatically journaled, so you can easily go back, if and when you need to write-up or revisit a particular experiment?

I'm interested both in what is considered the best way... but also, what people you actually do, which might be better than the so-called "best" (which might be too rigid, adding complexity overhead).

EDIT We have limited attention, and code, paramaters, experiment and hypotheses can be complex enough, without unnecessary management overhead - especially if you're still learning (as I am). So, quick-and-dirty sometimes can be better trade-off, in practice, taking into account the true context and needs, than a "proper" heavy-weight industrial/institutional/enterprise grade solution.

• You may want to consider looking into the field of "Research Data Management", of which your question is a subset. Note that since repeatability is one of the fundamentals of the scientific methods, you ideally wouldn't just store the configuration for your software, but version information, the source code and ideally even the hardware if this is is liable to change. – origimbo Jun 28 '17 at 10:44
• I agree it's a subset, but RDM seems very institution-oriented, and at a more abstract level than my simple and low-level needs. It's important if you're at a uni (but I'm not). The RDM details I did find were common-sense (e.g. Cambridge's: data.cam.ac.uk/data-management-guide/organising-your-data). I agree "repeatability" is important for science, but I'm at the stage of just trying to get something working, and want to record to help me understand what my code is doing (including debugging). e.g. just started Barba's "12 steps", and playing with parameters to see CFL blow-ups. – hyperpallium Jul 5 '17 at 8:20
• @origimbo PS of course, it never hurts to be reminded of common-sense! And RDM standardizes terminology for those concepts. – hyperpallium Jul 6 '17 at 4:14
• For your problem with storing input expressions in a file using a compiled languages, you could use an expression parser library, e.g., exprtk (assuming you use C++). – cfh Jul 10 '17 at 8:44
• @cfh Thanks; I'm using java, but I see that that library does diff/integration (like sympy), so would also be useful for method of manufactured solutions. For tracking code in general in a compiled language, I thought about invoking the compiler at runtime; but for many reasons, it seems simpler to represent code with code (e.g. "in a class"). – hyperpallium Jul 10 '17 at 9:03

Since you've asked about experiences, I've seen all three of the methods you suggest used. However best practise is that inputs to the code, and the code framework itself are kept separate. This means that only parameters you don't expect to ever need to change should be hard coded into your source. For example, if you were modelling the flight of cannonballs, you might hard code the value for gravitational acceleration, but not the radius of the cannonball, which forms part of the metadata for the individual simulation. That way the code base in your version control system doesn't get polluted with input data, and remains sane for someone else (or you six months later) to understand.

In terms of classes, versus configuration files for the simulation input data, that really comes down to the language being used and to whether you're developing an executable or a framework. Again, best practise is to use a self-documenting system like xml or JSON for configuration files (no need to roll your own implementation, there are prebuilt parsing libraries out there). Ideally the file should be human readable, either directly or via a helper utility. For the class based method, the attribute names should again be human understandable, although there are still people out there who are stuck in the FORTRAN 66 structure of six-letter variable names like "GRNEW4".

Outputting should again be in a commonly used, self-describing file format, and ideally should include enough metadata to be able to reproduce the original input data which generated it if necessary. Since input files tend to be small compared to the output data, one option is to simply prepend the input file/source at the start of the output data.

The method of organisation for the final output is probably the point of most debate. While version control systems work pretty well for input files (in a separate repository from the source code remember), they don't always play that well with binary data. This leads to a lot of disk based organisation (use human understandable file names!), but a useful possibility is to take advantage of one of the various generic reference managers like JabRef, which are readily available and provide searchable tracking of the simulation metadata.

• Which do you do? (I edited the question to clarify that that's what I meant by in your experience) – hyperpallium Jul 10 '17 at 4:45
• We have a limited amount of attention, and when experimenting with different parameters over complex code that we're trying to understand, we mightn't have any to spare for the extra complexity and overhead of "proper" engineering. That's why I wondered about the (bad practice hack) of commenting out code - it can be a good trade-off when attention is the limiting resource. Attention is further limited when learning the material - as I am. Here's an interesting hackernews discussion on academic vs professional coding that's somewhat relevant. – hyperpallium Jul 10 '17 at 4:59
• All good, sensible ideas! Having a separate data repository is intriguing, and would be helpful for distributing it (e.g. on github, BB), and could also track changes in the dataset itself (though one would hope the input data wouldn't be modified...). But code and data can be in the same repository, provided you never modify the data (If you need to tweak some data, copy and vary the copy.). Just use separate directory structures, for intelligibility. Thus, the input data will be available to all future versions of the code (though not to past versions... which might be a consideration). – hyperpallium Jul 10 '17 at 8:51

I don't know what you can do to keep track of parameters and results, though @origimbo correctly noted that repeatability kinda makes results useless to store.

The first point you make is obviously the versioning problem, so the solution is trivial: create a repository on Github or Bitbucket or Mercurial. These services are very effective and they let you reverse a given commit if you messed up your code and you need to restore it. They are supported as plugin within most IDEs.

Plus: redundancy on cloud! Making backup copies will be a bit less urgent than it usually is.

Edit: implement a logger into your programs.

• I do use BB as semi-backup, but using commits to store parameters is problematic: if you've developed your code further, and you checkout previous parameters, those code changes aren't available... you can merge the previous parameters (and previous code) with present code, but you must manually merge... Git commits are perfect to archive the expt together the exact code version but on-going development needs orthogonal param storage. I understand regenerating results means they needn't be recorded, but for me, they document the parameters' meaning (e.g. onset of instability in a plot). – hyperpallium Jul 5 '17 at 8:44
• @hyperpallium I don't quite understand what is your problem specifically. Can you make a detailed example of it? At this point my suggestion is: make your program LOG everything. – GoGoLander Jul 5 '17 at 11:26
• @hyperpallium you can log whatever you want in every way you want: you are in charge, you are implementing the logger, not the other way around. On that point, I don't see why you would log informations that you deem useless – GoGoLander Jul 5 '17 at 14:08
• I'm very lucky compared to you: I only have a couple of free parameters in my code so it is far simpler to track my changes. In general, every time I change something, I do it in a new branch of the repo: if the thing blows apart, I simply change the branch and close the messed up one. In closed branches I specify what went wrong and why they underperformed compared to other approches. It could look messy, but I actually stopped making the same mistakes again and again and I'm making real progress. – GoGoLander Jul 10 '17 at 9:40
• I forgot the most important part: the good branches, i.e. the ones with better results and better performances, are then consolidated in the official branch – GoGoLander Jul 10 '17 at 9:48

Manage parameters with switch statements: each case is a self-contained set of parameters. This doesn't address different expressions/codes etc, but for Barba's 12 steps, it's enough so far. EDIT and arbitrary boolean switches for different parts of the code (e.g. different boundary conditions; switching off a stage such as advection). Messy but flexible.

Add parameters to output plots: so the input and interpretable results are documented together, in an image file.

Automatically timestamping the plot images, and outputting to a tmp directory, so I can go back to them if need be, (and copy and rename any I want to keep). This is the journalling idea; and, I guess is a form of "logging".

I'm using gnuplot for plotting, and send in the parameters and filename as "call" arguments (this line is auto-generated from my simulation code):

gnuplot -c myscript filename stringofparameternamesandvalues


These argumemts are available as ARG1 and ARG2 in the script. In the script, I set the output file and title to them. One can also put them in a label, and position it (and use \n for multiline), but I'm concerned it will sometimes overwrite part of the plot - using the title prevents that.

Thought: a parameter object. Using reflection, one can automatically serialize to JSON (or XML), or output to a logger (as @GoGoLander suggested). An advantage is if you add a new parameter to the object, you don't have to remember to add it the output code (the serializer automatically output all the object's fields). Though this doesn't capture expressions/code. EDIT a parameter object is verbose because must add prefix (e.g. params.dx). Instead, use fields on the code's own instance object as parameters, and can use them directly (e.g. dx). Here's a way to auto-output them in java (assuming no other fields):

import java.lang.reflect.Field;
...
public String getParams() {
String a = "";
try {
for (Field f : this.getClass().getDeclaredFields()) {
f.setAccessible(true);  // so needn't make fields public
a += f.getName() + "="+ f.get(this) +", ";
}
} catch (Exception e) { e.printStackTrace(); }
return a;
}


I think "2. in a class" is the right approach for compiled languages, because it can store expressions and code, and keeps experimental config separate from the main code.