How to separate a solver from computational experiments in a correct way?

Question

I do computational work as a PhD student and I try to find a correct way to separate code (solver) from the computational experiments based on this solver.

Basically, each project that I do revolves around one particular solver (MATLAB or Python code), and for each project there is a set of experiments that should be done. Before, I came to the idea to do it in the following way:

project
    01-2016-01-01-task-1
    02-2016-01-01-task-2
    03-2016-01-03-task-3

such that each subfolder corresponds to one particular task (or experiment) that I do. Each subfolder contains all the output from the solver and the solver code itself as I frequently have to modify the solver for a particular experiment.

I've already deduced that the best way to modify the solver code is in a backward-compatible way, such that if an experiment requires that I change the code to use, for example, a different time integrator (I solve PDEs), then a new configuration parameter should be added to the solver which defines what time integrator to use in the runtime.

However, I have a problem with this approach (keeping local version of solver for each experiment): I have to merge all these different versions together and I do it manually now. Recently, an idea came to me that a better organization is

project
    code
    tasks
        01-2016-01-01-task-01
        ...
        27-2016-03-06-task-27

that is, that the code should be centralized and version controlled.

Now imagine the following situation. Experiment $N$ requires that I modify the code and then run solver three times with different parameters and each run takes 20 minutes. I do it with a bash script which runs the solver three times sequentially. In the meantime, I start experiment $N+1$ which also requires code modification. I start to modify the code and it can break the experiment $N$ (for example, between run 1 and run 2).

How do you handle such situations? Is it better to fork the solver code for each experiment and keep it under the experiment subfolder?

UPDATED 2016-09-28: According to the answers and comments that I got, it seems that the question was not written clear enough, particularly the title.

My question was about how to conduct experiments based on a simulation code, where the code must evolve simultaneously with the progress of the project. Indeed, usually, the code admits some generality from the very start (for example, all problem parameters can be specified as command-line arguments, as well as some solver parameters). However, it is quite possible that the code will require further generalization after a series of experiments, for example, one finite-difference approximation should be replaced by another. In this case, as many pointed out below, the best solution is to add an option to the solver, which chooses the needed approximation during the runtime.

Therefore, it is better to version-control it as well as to isolate a copy of the code for each experiment. This is what my question was about: how to deal with multiple copies of the code for each experiment?

If you don't use any version-control system, you can just copy the code into the subdirectory of each experiment and then manually merge any modifications to the "golden" copy of the code (this is my previous approach).

If you use version-control system Git, you can make multiple clones of the repository to have isolated copies. However, as Git 2.5+ provides a feature to have multiple working copies of the repository, the answer that I wrote below proposes more elegant approach.

Answer 1

While Greg Wilson's publications in @nicoguaro's answer give great advice, I also think that the situation you describe would best be handled by giving your solver options for runtime configuration. This article by Jed Brown (also a SciComp user) describes some good high-level practices, and you can also see examples of runtime configuration using PETSc in http://www.mcs.anl.gov/uploads/cels/papers/P2010-0112.pdf.

The basic idea is that if you're changing parameters, these should all be runtime options, so you can run your code with ./mycode -parameter_a 1.7 -parameter_b 2.5 and change the parameters at runtime again with ./mycode -parameter_a 2.6 -parameter_b 3.4. In an ideal world, you should never have to recompile your code to run experiments. In practice, of course, setting up runtime options requires some infrastructure, but if you can do it, the investment is well worth the effort because it will make running concurrent experiments with different parameter values easy.

Answer 2

Thanks to everyone for their comments. I've found the following good solution to my question. Apparently, each task should have its own version of the code to isolate tasks from each other and to ensure reproducibility in a sense that it must be always clear what version of the code was used for the task. The remaining question was: how to avoid cloning a git repository multiple times?

Git version 2.5+ allows creating multiple worktrees connected to the same repository. Using multiple worktrees, you can avoid cloning the repository multiple times, and at the same time track all versions of the code in separate branches.

So, the structure of the project will be like this:

project
    code
    tasks
        01-2016-01-01-task-01
            code
        ...
        27-2016-03-06-task-27
            code

where each directory 01-2016-01-01-task-01 and similar will have its own copy of the code set to its own branch.

To create additional worktrees, the following git command is used:

cd <project>/code
git worktree add -b 01-2016-01-01-task-01 \
    ../tasks/01-2016-01-01-task-01/code master

which creates a branch 01-2016-01-01-task-01 from master and checks it out into the directory ../tasks/01-2016-01-01-task-01/code.

Then, when the work on the given task is finished, you can merge the corresponding branch into the master branch.

Answer 3

Part of your question was addressed in a recent publication "Good Enough Practices in Scientific Computing" (reference 1), primarily in point 4. I added the whole list, though.

1. Data Management
   1. Save the raw data.
   2. Create the data you wish to see in the world
   3. Create analysis-friendly data.
   4. Record all the steps used to process data.
   5. Anticipate the need to use multiple tables.
   6. Submit data to a reputable DOI-issuing repository so that others can access and cite it.
2. Software
   1. Place a brief explanatory comment at the start of every program.
   2. Decompose programs into functions.
   3. Be ruthless about eliminating duplication.
   4. Always search for well-mantained software libraries that do what you need.
   5. Test libraries before relying on them.
   6. Give functions and variables meaningful names.
   7. Make dependecies and requirements explicit.
   8. Do not comment and uncomment sections of code to control a program's behavior.
   9. Provide a simple example or test data set.
   10. Submit code to a reputable DOI-issuing repository.
3. Collaboration
   1. Create an overview of your project.
   2. Create a shared public "to-do" list.
   3. Make the license explicit.
   4. Make the project citable.
4. Project Organization
   1. Put each project in its own directory, which is name after the project.
   2. Put text documents associated with the project in the doc directory.
   3. Put raw data and metadata in a data directory, and files generated during cleanup and analysis in a results directory.
   4. Put project source code in the src directory.
   5. Put external scripts, or compiled programs in the bin directory.
   6. Name files to reflect their content or function.
5. Keeping Track of Changes
   1. Back up (almost) everything created by a human as soon as it is created.
   2. Keep changes small.
   3. Share changes frequently.
   4. Create, mantain, and use a checklist for saving and sharing changes to the project.
   5. Store each project in a folder that is mirrored off the the reasearcher's working machine.
   6. Add a file called CHANGELOG.txt to the project's docs subfolder.
   7. Copy the entire project whenever a significant change has been made.
6. Manuscripts
   1. Write manuscripts using online tools with rich formatting, change tracking, and reference management.
   2. Include a PUBLICATIONS file in the project's doc directory.
   3. Write the manuscript in a plain text format that permits version control.

References

1. Wilson, Greg, Jennifer Bryan, Karen Cranston, Justin Kitzes, Lex Nederbragt, and Tracy K. Teal. "Good Enough Practices in Scientific Computing." arXiv preprint arXiv:1609.00037 (2016)

2. Wilson, Greg, D. A. Aruliah, C. Titus Brown, Neil P. Chue Hong, Matt Davis, Richard T. Guy, Steven HD Haddock et al. "Best practices for scientific computing." PLoS Biol 12, no. 1 (2014): e1001745.