7
$\begingroup$

I do computational work as a PhD student and I try to find a correct way to separate code (solver) from the computational experiments based on this solver.

Basically, each project that I do revolves around one particular solver (MATLAB or Python code), and for each project there is a set of experiments that should be done. Before, I came to the idea to do it in the following way:

project
    01-2016-01-01-task-1
    02-2016-01-01-task-2
    03-2016-01-03-task-3

such that each subfolder corresponds to one particular task (or experiment) that I do. Each subfolder contains all the output from the solver and the solver code itself as I frequently have to modify the solver for a particular experiment.

I've already deduced that the best way to modify the solver code is in a backward-compatible way, such that if an experiment requires that I change the code to use, for example, a different time integrator (I solve PDEs), then a new configuration parameter should be added to the solver which defines what time integrator to use in the runtime.

However, I have a problem with this approach (keeping local version of solver for each experiment): I have to merge all these different versions together and I do it manually now. Recently, an idea came to me that a better organization is

project
    code
    tasks
        01-2016-01-01-task-01
        ...
        27-2016-03-06-task-27

that is, that the code should be centralized and version controlled.

Now imagine the following situation. Experiment $N$ requires that I modify the code and then run solver three times with different parameters and each run takes 20 minutes. I do it with a bash script which runs the solver three times sequentially. In the meantime, I start experiment $N+1$ which also requires code modification. I start to modify the code and it can break the experiment $N$ (for example, between run 1 and run 2).

How do you handle such situations? Is it better to fork the solver code for each experiment and keep it under the experiment subfolder?

UPDATED 2016-09-28: According to the answers and comments that I got, it seems that the question was not written clear enough, particularly the title.

My question was about how to conduct experiments based on a simulation code, where the code must evolve simultaneously with the progress of the project. Indeed, usually, the code admits some generality from the very start (for example, all problem parameters can be specified as command-line arguments, as well as some solver parameters). However, it is quite possible that the code will require further generalization after a series of experiments, for example, one finite-difference approximation should be replaced by another. In this case, as many pointed out below, the best solution is to add an option to the solver, which chooses the needed approximation during the runtime.

Therefore, it is better to version-control it as well as to isolate a copy of the code for each experiment. This is what my question was about: how to deal with multiple copies of the code for each experiment?

If you don't use any version-control system, you can just copy the code into the subdirectory of each experiment and then manually merge any modifications to the "golden" copy of the code (this is my previous approach).

If you use version-control system Git, you can make multiple clones of the repository to have isolated copies. However, as Git 2.5+ provides a feature to have multiple working copies of the repository, the answer that I wrote below proposes more elegant approach.

$\endgroup$
  • $\begingroup$ Do you mean to say the individual copies of solvers weren't version-controlled? It's probably worth looking at how tightly coupled the different components of the solver are, and whether they can be decoupled better, and this could just indicate a flaw in the program's design; see also strategy pattern, which may or may not be helpful to you. The issue with modifying running code is unusual, and could be solved with proper use of VC (tie each experiment to specific VC revision). $\endgroup$ – Kirill Sep 10 '16 at 4:14
  • $\begingroup$ How similar are your experiments? Will they be tied together in the end? I would suggest thinking about how much code, if any, you can reuse from each of these experiments, if the answer is: a lot, then I would suggest maintaining one project, and keeping this source code separate from the experiments (your second diagram). $\endgroup$ – Charles Sep 10 '16 at 5:57
  • $\begingroup$ @Kirill, yes, I decouple the components of my code using strategy pattern. How to tie an experiment to a particular VC version (I use git) in an optimal way? I'd like the scheme to be as simple and transparent as it can be. $\endgroup$ – Dmitry Kabanov Sep 10 '16 at 10:52
  • $\begingroup$ @Charlie, this is exactly what I'm trying to do. The question is how to handle code modifications for several experiments running simulatenously such that the earlier started experiment doesn't break during execution. $\endgroup$ – Dmitry Kabanov Sep 10 '16 at 10:54
  • $\begingroup$ I'm not an expert in this area, but I would pipeline the 1) build and run process and 2) experiment test runs. This way, after you make changes you can build, run and check if the change affects your previously benchmarked experiments. $\endgroup$ – Charles Sep 10 '16 at 16:40
3
$\begingroup$

While Greg Wilson's publications in @nicoguaro's answer give great advice, I also think that the situation you describe would best be handled by giving your solver options for runtime configuration. This article by Jed Brown (also a SciComp user) describes some good high-level practices, and you can also see examples of runtime configuration using PETSc in http://www.mcs.anl.gov/uploads/cels/papers/P2010-0112.pdf.

The basic idea is that if you're changing parameters, these should all be runtime options, so you can run your code with ./mycode -parameter_a 1.7 -parameter_b 2.5 and change the parameters at runtime again with ./mycode -parameter_a 2.6 -parameter_b 3.4. In an ideal world, you should never have to recompile your code to run experiments. In practice, of course, setting up runtime options requires some infrastructure, but if you can do it, the investment is well worth the effort because it will make running concurrent experiments with different parameter values easy.

$\endgroup$
  • $\begingroup$ I've read the article by Jed Brown last year, thank you. It actually influenced me quite a lot such that I started to use the strategy pattern more intensely to make some components of the code replaceable. But my question was on a different topic. I should definitely edit it to emphasize what I'm asking about. $\endgroup$ – Dmitry Kabanov Sep 24 '16 at 16:05
  • $\begingroup$ I don't see edits to your question yet, but in general, the best solution is to structure your code so that changes you make for experiments will be runtime commands. If you can't make these changes without changing the code for whatever reason, I see no harm in using separate commits for experiments with different parameters. Sure, it makes the repo messier, and it's not the "cleanest" approach to purists, but tracking file changes is the purpose of VCSes. As for the output of the code, that's up to you; you could version the script that generates the output, or also track the output itself. $\endgroup$ – Geoff Oxberry Sep 26 '16 at 21:32
1
$\begingroup$

Thanks to everyone for their comments. I've found the following good solution to my question. Apparently, each task should have its own version of the code to isolate tasks from each other and to ensure reproducibility in a sense that it must be always clear what version of the code was used for the task. The remaining question was: how to avoid cloning a git repository multiple times?

Git version 2.5+ allows creating multiple worktrees connected to the same repository. Using multiple worktrees, you can avoid cloning the repository multiple times, and at the same time track all versions of the code in separate branches.

So, the structure of the project will be like this:

project
    code
    tasks
        01-2016-01-01-task-01
            code
        ...
        27-2016-03-06-task-27
            code

where each directory 01-2016-01-01-task-01 and similar will have its own copy of the code set to its own branch.

To create additional worktrees, the following git command is used:

cd <project>/code
git worktree add -b 01-2016-01-01-task-01 \
    ../tasks/01-2016-01-01-task-01/code master

which creates a branch 01-2016-01-01-task-01 from master and checks it out into the directory ../tasks/01-2016-01-01-task-01/code.

Then, when the work on the given task is finished, you can merge the corresponding branch into the master branch.

$\endgroup$
0
$\begingroup$

Part of your question was addressed in a recent publication "Good Enough Practices in Scientific Computing" (reference 1), primarily in point 4. I added the whole list, though.

  1. Data Management
    1. Save the raw data.
    2. Create the data you wish to see in the world
    3. Create analysis-friendly data.
    4. Record all the steps used to process data.
    5. Anticipate the need to use multiple tables.
    6. Submit data to a reputable DOI-issuing repository so that others can access and cite it.
  2. Software
    1. Place a brief explanatory comment at the start of every program.
    2. Decompose programs into functions.
    3. Be ruthless about eliminating duplication.
    4. Always search for well-mantained software libraries that do what you need.
    5. Test libraries before relying on them.
    6. Give functions and variables meaningful names.
    7. Make dependecies and requirements explicit.
    8. Do not comment and uncomment sections of code to control a program's behavior.
    9. Provide a simple example or test data set.
    10. Submit code to a reputable DOI-issuing repository.
  3. Collaboration
    1. Create an overview of your project.
    2. Create a shared public "to-do" list.
    3. Make the license explicit.
    4. Make the project citable.
  4. Project Organization
    1. Put each project in its own directory, which is name after the project.
    2. Put text documents associated with the project in the doc directory.
    3. Put raw data and metadata in a data directory, and files generated during cleanup and analysis in a results directory.
    4. Put project source code in the src directory.
    5. Put external scripts, or compiled programs in the bin directory.
    6. Name files to reflect their content or function.
  5. Keeping Track of Changes
    1. Back up (almost) everything created by a human as soon as it is created.
    2. Keep changes small.
    3. Share changes frequently.
    4. Create, mantain, and use a checklist for saving and sharing changes to the project.
    5. Store each project in a folder that is mirrored off the the reasearcher's working machine.
    6. Add a file called CHANGELOG.txt to the project's docs subfolder.
    7. Copy the entire project whenever a significant change has been made.
  6. Manuscripts
    1. Write manuscripts using online tools with rich formatting, change tracking, and reference management.
    2. Include a PUBLICATIONS file in the project's doc directory.
    3. Write the manuscript in a plain text format that permits version control.

References

  1. Wilson, Greg, Jennifer Bryan, Karen Cranston, Justin Kitzes, Lex Nederbragt, and Tracy K. Teal. "Good Enough Practices in Scientific Computing." arXiv preprint arXiv:1609.00037 (2016)

  2. Wilson, Greg, D. A. Aruliah, C. Titus Brown, Neil P. Chue Hong, Matt Davis, Richard T. Guy, Steven HD Haddock et al. "Best practices for scientific computing." PLoS Biol 12, no. 1 (2014): e1001745.

$\endgroup$
  • 1
    $\begingroup$ I've read the ref. 1, thanks for sharing it. Somehow, the authors do not distinguish between projects and experiments. In my view, a project in computational science consists of a set of experiments. So, the directory structure that I came to is more suitable for me. $\endgroup$ – Dmitry Kabanov Sep 24 '16 at 16:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.