I do computational work as a PhD student and I try to find a correct way to separate code (solver) from the computational experiments based on this solver.
Basically, each project that I do revolves around one particular solver (MATLAB or Python code), and for each project there is a set of experiments that should be done. Before, I came to the idea to do it in the following way:
project 01-2016-01-01-task-1 02-2016-01-01-task-2 03-2016-01-03-task-3
such that each subfolder corresponds to one particular task (or experiment) that I do. Each subfolder contains all the output from the solver and the solver code itself as I frequently have to modify the solver for a particular experiment.
I've already deduced that the best way to modify the solver code is in a backward-compatible way, such that if an experiment requires that I change the code to use, for example, a different time integrator (I solve PDEs), then a new configuration parameter should be added to the solver which defines what time integrator to use in the runtime.
However, I have a problem with this approach (keeping local version of solver for each experiment): I have to merge all these different versions together and I do it manually now. Recently, an idea came to me that a better organization is
project code tasks 01-2016-01-01-task-01 ... 27-2016-03-06-task-27
that is, that the code should be centralized and version controlled.
Now imagine the following situation. Experiment $N$ requires that I modify the code and then run solver three times with different parameters and each run takes 20 minutes. I do it with a bash script which runs the solver three times sequentially. In the meantime, I start experiment $N+1$ which also requires code modification. I start to modify the code and it can break the experiment $N$ (for example, between run 1 and run 2).
How do you handle such situations? Is it better to fork the solver code for each experiment and keep it under the experiment subfolder?
UPDATED 2016-09-28: According to the answers and comments that I got, it seems that the question was not written clear enough, particularly the title.
My question was about how to conduct experiments based on a simulation code, where the code must evolve simultaneously with the progress of the project. Indeed, usually, the code admits some generality from the very start (for example, all problem parameters can be specified as command-line arguments, as well as some solver parameters). However, it is quite possible that the code will require further generalization after a series of experiments, for example, one finite-difference approximation should be replaced by another. In this case, as many pointed out below, the best solution is to add an option to the solver, which chooses the needed approximation during the runtime.
Therefore, it is better to version-control it as well as to isolate a copy of the code for each experiment. This is what my question was about: how to deal with multiple copies of the code for each experiment?
If you don't use any version-control system, you can just copy the code into the subdirectory of each experiment and then manually merge any modifications to the "golden" copy of the code (this is my previous approach).
If you use version-control system Git, you can make multiple clones of the repository to have isolated copies. However, as Git 2.5+ provides a feature to have multiple working copies of the repository, the answer that I wrote below proposes more elegant approach.