# Scientific workflow management system

Can anyone recommend me a good workflow management system (WMS), preferably in Python? So far I have been using GNU Make, but it introduces a layer of complexity that I want to avoid. A good WMS should have the following features:

• integrate easily with command line tools and Python scripts,
• simple to use and lightweight,
• handle dependencies,
• provide command line interface,
• provide logging mechanism,
• (optional) provide data provenance.

I know that WMS are very popular in bioinformatics (for example Galaxy), but I am looking for something more general.

-
This isn't a complete answer, but since you mentioned GNU Make and Python in the same question, thought I would point you towards SCons: scons.org –  Reid.Atcheson Mar 13 '12 at 16:02
Thanks. Do you know of any examples using SCons for scientific workflows? –  btel Mar 13 '12 at 16:42
I've found that with a little bit of work, you can get emacs to do most of that (sometimes by integrating with outside tools). It's probably not what you're looking for, though, since I find I generally still have to use makefiles to compile anything nontrivial. –  Dan Mar 13 '12 at 17:13
I can write an answer about SCons and waf, which are Python build system tools. I've used SCons for a few months now, and can give you some perspective on what's good and bad about it relative to GNU Make. That said, I was wondering if you could elaborate on what you mean by "provide logging mechanism" and "provide data provenance". In terms of logging, do you just want a logfile, or are you looking for something more like a version control system? –  Geoff Oxberry Mar 13 '12 at 18:25
Logging might be something very simple like logfiles, as you suggest, with timestamps of all runs, redirect of stderr and (optionally) stdout. In addition one could keep the intermediate results from each step in the workflow in a separate directory. Data provenance is something more like a control version system that keeps the history of all computation scripts, input and output files. Currently, I use Makefiles + git but I look for something better integrated and easier to use. I have heard of SCons but I do not know what is its advantage over Make. –  btel Mar 14 '12 at 10:22

For logging that allows full reproducibility, I highly recommend the Sumatra python package. It nicely links the version control commit number, machine state, and output files to each program run and has a django web interface to interact with the database of run info. The python API makes it very easy to include logging in my scripts.

-
Sumatra looks really interesting; I'll have to give it a try. –  Geoff Oxberry Apr 11 '12 at 4:03
It does not fulfill all of my requirements, but it is closest to what I need. Therefore, I accepted the answer. Disclosure: I am one of developers of sumatra. –  btel Nov 19 '12 at 14:16

Some month ago, I stumbled upon the highly recommended website of Hans-Martin v. Gaudecker who teaches courses like "Effective programming practices for economists". In his Autumn 2010 course he introduced SCons, in his Autumn 2011 course he switched to waf, which is supposed to be faster than SCons but still Python-based. The slides for both courses are available for download and I (as a social scientist) found them very instructive and enlightening.

-
SCons is pretty rad. It supports very complicated or very simple schemae equally well! –  meawoppl Mar 15 '12 at 21:00
The tradeoff between SCons and any faster build tool generally has to do with dependency checking. For mainstream languages (C, C++, Fortran, D, Python, Java, etc.), SCons will automatically determine dependencies using an MD5 hash-based algorithm, rather than time stamps, which can be fragile when dealing with generated files. Everything else beats SCons in performance (time needed to build software) because they don't do as much dependency checking, or they offload the dependency checking to some other tool (like the compilers used). –  Geoff Oxberry Mar 18 '12 at 0:57
The first link of your answer is 404 now. It seems that his new page is at uni-bonn.de/~hmg308/teaching.html –  liori Aug 20 '12 at 14:33
@liori fixed, thanks! –  Bernd Weiss Aug 21 '12 at 5:34
SCons has configurable "up-to-dateness" checking, so you can choose between timestamp, hash, or some combination. That said, I'm growing disenchanted with it: A few things are very easy (e.g. compiling software using a tool chain for which SCons has good Tool packages) and almost anything is possible, but it gets ugly pretty quickly. –  Eric Anderson Feb 24 at 17:13

Take a look at VisTrails. I haven't used it (only homebrew stuff around make), but it looks well thought-out, with good doc, and has real users at NASA etc.
(Are you looking for tools for 1-2 people, 4-5, more ?)

for uniform, reproducible computer experiments one obviously needs

• uniform directory structures, e.g. when-what/ in/ out/ scripts/ log/
• uniform setting and echoing of all parameters for a run
• scripts to summarize / plot / evaluate runs.

See also software-carpentry.org: "The problem we’re trying to solve is that scientists often spend 40% or more of their time wrestling with software, but 95% or more of them are primarily self-taught".

-

All the requirements you mentioned in your question are fulfilled by the Swift parallel scripting system.

I've spent a year with Swift group as a postdoctoral researcher (PhD in scientific workflows). We've been helping scientists and researchers from different domains address their computational needs.

Swift is an open source framework for running workflows in parallel manner. It is called parallel scripting mainly to highlight the fact that it provides a scripting interface to creating workflows as opposed to the GUI box-arrow interface.

-
Welcome to scicomp! Do you mind expanding your answer a little more (click the little gray edit button below your answer) to edit. Also, can you make your connection to Swift a little more clear in your answer? Thanks! –  Aron Ahmadia Nov 10 '12 at 22:36
I expanded my answer! Thanks. –  Ketan Nov 10 '12 at 22:42

Taverna is an open-source WMS, not Python but Java.

-
Have you used it? –  Deathbreath Mar 13 '12 at 20:53
Thanks for the suggestion. I saw the Taverna website, but it looks like a mainly graphical tool. I rather look for something command-line-based. Taverna does provide command-line tool, but it is only to execute workflows, but not build them (is it correct?). It also seems very much bioinformatics-oriented. –  btel Mar 14 '12 at 10:31
It seems to me, you're more looking for a LIMS suitable for numerical experiments, rather than a build system like make or scons? –  GertVdE Mar 15 '12 at 8:15
Sorry to ask. What does LIMS stand for exactly? –  btel Mar 15 '12 at 14:38
Laboratory Information Management System. It's a family of tools to keep log of lab experiments. But these are typically, for example, for chemical analyses. You might want to Google for "in silico experiments", i.e. experiments that are simulations on a computer and require "logging" -> storing input/output data, what version of the software was used, hypotheses, ... –  GertVdE Mar 15 '12 at 19:39