# Scientific workflow management system

Can anyone recommend me a good workflow management system (WMS), preferably in Python? So far I have been using GNU Make, but it introduces a layer of complexity that I want to avoid. A good WMS should have the following features:

• integrate easily with command line tools and Python scripts,
• simple to use and lightweight,
• handle dependencies,
• provide command line interface,
• provide logging mechanism,
• (optional) provide data provenance.

I know that WMS are very popular in bioinformatics (for example Galaxy), but I am looking for something more general.

• This isn't a complete answer, but since you mentioned GNU Make and Python in the same question, thought I would point you towards SCons: scons.org – Reid.Atcheson Mar 13 '12 at 16:02
• Thanks. Do you know of any examples using SCons for scientific workflows? – btel Mar 13 '12 at 16:42
• I've found that with a little bit of work, you can get emacs to do most of that (sometimes by integrating with outside tools). It's probably not what you're looking for, though, since I find I generally still have to use makefiles to compile anything nontrivial. – Dan Mar 13 '12 at 17:13
• I can write an answer about SCons and waf, which are Python build system tools. I've used SCons for a few months now, and can give you some perspective on what's good and bad about it relative to GNU Make. That said, I was wondering if you could elaborate on what you mean by "provide logging mechanism" and "provide data provenance". In terms of logging, do you just want a logfile, or are you looking for something more like a version control system? – Geoff Oxberry Mar 13 '12 at 18:25
• Logging might be something very simple like logfiles, as you suggest, with timestamps of all runs, redirect of stderr and (optionally) stdout. In addition one could keep the intermediate results from each step in the workflow in a separate directory. Data provenance is something more like a control version system that keeps the history of all computation scripts, input and output files. Currently, I use Makefiles + git but I look for something better integrated and easier to use. I have heard of SCons but I do not know what is its advantage over Make. – btel Mar 14 '12 at 10:22

For logging that allows full reproducibility, I highly recommend the Sumatra python package. It nicely links the version control commit number, machine state, and output files to each program run and has a django web interface to interact with the database of run info. The python API makes it very easy to include logging in my scripts.

• Sumatra looks really interesting; I'll have to give it a try. – Geoff Oxberry Apr 11 '12 at 4:03
• It does not fulfill all of my requirements, but it is closest to what I need. Therefore, I accepted the answer. Disclosure: I am one of developers of sumatra. – btel Nov 19 '12 at 14:16

Some month ago, I stumbled upon the highly recommended website of Hans-Martin v. Gaudecker who teaches courses like "Effective programming practices for economists". In his Autumn 2010 course he introduced SCons, in his Autumn 2011 course he switched to waf, which is supposed to be faster than SCons but still Python-based. The slides for both courses are available for download and I (as a social scientist) found them very instructive and enlightening.

• SCons is pretty rad. It supports very complicated or very simple schemae equally well! – meawoppl Mar 15 '12 at 21:00
• The tradeoff between SCons and any faster build tool generally has to do with dependency checking. For mainstream languages (C, C++, Fortran, D, Python, Java, etc.), SCons will automatically determine dependencies using an MD5 hash-based algorithm, rather than time stamps, which can be fragile when dealing with generated files. Everything else beats SCons in performance (time needed to build software) because they don't do as much dependency checking, or they offload the dependency checking to some other tool (like the compilers used). – Geoff Oxberry Mar 18 '12 at 0:57
• The first link of your answer is 404 now. It seems that his new page is at uni-bonn.de/~hmg308/teaching.html – liori Aug 20 '12 at 14:33
• SCons has configurable "up-to-dateness" checking, so you can choose between timestamp, hash, or some combination. That said, I'm growing disenchanted with it: A few things are very easy (e.g. compiling software using a tool chain for which SCons has good Tool packages) and almost anything is possible, but it gets ugly pretty quickly. – Eric Anderson Feb 24 '14 at 17:13

Take a look at VisTrails. I haven't used it (only homebrew stuff around make), but it looks well thought-out, with good doc, and has real users at NASA etc.
(Are you looking for tools for 1-2 people, 4-5, more ?)

for uniform, reproducible computer experiments one obviously needs

• uniform directory structures, e.g. when-what/ in/ out/ scripts/ log/
• uniform setting and echoing of all parameters for a run
• scripts to summarize / plot / evaluate runs.

See also software-carpentry.org: "The problem we’re trying to solve is that scientists often spend 40% or more of their time wrestling with software, but 95% or more of them are primarily self-taught".

All the requirements you mentioned in your question are fulfilled by the Swift parallel scripting system.

I've spent a year with Swift group as a postdoctoral researcher (PhD in scientific workflows). We've been helping scientists and researchers from different domains address their computational needs.

Swift is an open source framework for running workflows in parallel manner. It is called parallel scripting mainly to highlight the fact that it provides a scripting interface to creating workflows as opposed to the GUI box-arrow interface.

• Welcome to scicomp! Do you mind expanding your answer a little more (click the little gray edit button below your answer) to edit. Also, can you make your connection to Swift a little more clear in your answer? Thanks! – Aron Ahmadia Nov 10 '12 at 22:36

Taverna is an open-source WMS, not Python but Java.

• Have you used it? – Deathbreath Mar 13 '12 at 20:53
• Thanks for the suggestion. I saw the Taverna website, but it looks like a mainly graphical tool. I rather look for something command-line-based. Taverna does provide command-line tool, but it is only to execute workflows, but not build them (is it correct?). It also seems very much bioinformatics-oriented. – btel Mar 14 '12 at 10:31
• It seems to me, you're more looking for a LIMS suitable for numerical experiments, rather than a build system like make or scons? – GertVdE Mar 15 '12 at 8:15
• Sorry to ask. What does LIMS stand for exactly? – btel Mar 15 '12 at 14:38
• Laboratory Information Management System. It's a family of tools to keep log of lab experiments. But these are typically, for example, for chemical analyses. You might want to Google for "in silico experiments", i.e. experiments that are simulations on a computer and require "logging" -> storing input/output data, what version of the software was used, hypotheses, ... – GertVdE Mar 15 '12 at 19:39

Another promising package not mentioned here is Snakemake.

Dexy sounds like it is exactly what you area after. From the site:

Dexy is a multi-purpose project automation tool with lots of features designed for working with documents. Dexy is written in Python and has a command-line interface. It's open source software with an MIT license.

### What does Dexy do?

Dexy makes it easier to create technical documents by doing the repetitive parts for you. Dexy provides a consistent interface to tools and scripts so you don't have to run them manually. Your project's dexy configuration keeps track of what to run, in which order, and with what parameters. This way, your whole process is captured so anyone can run it using one simple command and the results will be consistent.

You want to write a blog post with examples showing how to use an API. Dexy will automatically:

• run your example code, saving the results
• apply syntax highlighting to your example code (using pygments)
• insert the results of API calls and your prettified example code into your post (using jinja)
• convert your markdown-formatted blog post to HTML (using python markdown or pandoc).
• upload the HTML to the WordPress API in draft mode (using the WordPress API)
• publish your blog post when you are finished tweaking it
• I've followed Dexy for a few years, and the impressions I've gotten are that it's not widely adopted, and it's not actively developed. These traits could be a chicken-and-egg problem (a small user base means it's not actively developed, not being actively developed hurts user base growth). It looks super cool, and on it's face, I think it's exactly what scientists need to broaden reproducibility beyond IPython, knitr, and bespoke scripts, but for some reason...it just doesn't seem like it gets used. Ana Nelson doesn't even blog that much about it, and she wrote it. – Geoff Oxberry Aug 7 '15 at 4:55
• Well, the latest blob post is from January, and there have been 3 commits this year. Not super active, but not dead, especially if it's one of those project that just gets to stable and doesn't really need any more work. There are other project I use with much deader recent development histories. As for the chicken-and-egg problem, maybe an upvote here and wherever else it's mentioned on SE would help :P – naught101 Aug 11 '15 at 5:30

Knime or Rapidminer could likely handle all the requirements as well. Whether they should is up for grabs :)

• Maybe Kepler too. – Grimbly Mar 16 '16 at 23:52
• Also, Nifi is quite nifty, but I don't think it fully covers all of the requirements. – Grimbly Mar 17 '16 at 0:56