31
$\begingroup$

Can anyone recommend me a good workflow management system (WMS), preferably in Python? So far I have been using GNU Make, but it introduces a layer of complexity that I want to avoid. A good WMS should have the following features:

  • integrate easily with command line tools and Python scripts,
  • simple to use and lightweight,
  • handle dependencies,
  • provide command line interface,
  • provide logging mechanism,
  • (optional) provide data provenance.

I know that WMS are very popular in bioinformatics (for example Galaxy), but I am looking for something more general.

Improve this question
$\endgroup$
9
  • 2
    $\begingroup$ This isn't a complete answer, but since you mentioned GNU Make and Python in the same question, thought I would point you towards SCons: scons.org $\endgroup$
    – Reid.Atcheson
    Mar 13, 2012 at 16:02
  • $\begingroup$ Thanks. Do you know of any examples using SCons for scientific workflows? $\endgroup$
    – btel
    Mar 13, 2012 at 16:42
  • $\begingroup$ I've found that with a little bit of work, you can get emacs to do most of that (sometimes by integrating with outside tools). It's probably not what you're looking for, though, since I find I generally still have to use makefiles to compile anything nontrivial. $\endgroup$
    – Dan
    Mar 13, 2012 at 17:13
  • 1
    $\begingroup$ I can write an answer about SCons and waf, which are Python build system tools. I've used SCons for a few months now, and can give you some perspective on what's good and bad about it relative to GNU Make. That said, I was wondering if you could elaborate on what you mean by "provide logging mechanism" and "provide data provenance". In terms of logging, do you just want a logfile, or are you looking for something more like a version control system? $\endgroup$
    – Geoff Oxberry
    Mar 13, 2012 at 18:25
  • 1
    $\begingroup$ Logging might be something very simple like logfiles, as you suggest, with timestamps of all runs, redirect of stderr and (optionally) stdout. In addition one could keep the intermediate results from each step in the workflow in a separate directory. Data provenance is something more like a control version system that keeps the history of all computation scripts, input and output files. Currently, I use Makefiles + git but I look for something better integrated and easier to use. I have heard of SCons but I do not know what is its advantage over Make. $\endgroup$
    – btel
    Mar 14, 2012 at 10:22

8 Answers 8

Reset to default
12
$\begingroup$

For logging that allows full reproducibility, I highly recommend the Sumatra python package. It nicely links the version control commit number, machine state, and output files to each program run and has a django web interface to interact with the database of run info. The python API makes it very easy to include logging in my scripts.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Sumatra looks really interesting; I'll have to give it a try. $\endgroup$
    – Geoff Oxberry
    Apr 11, 2012 at 4:03
  • $\begingroup$ It does not fulfill all of my requirements, but it is closest to what I need. Therefore, I accepted the answer. Disclosure: I am one of developers of sumatra. $\endgroup$
    – btel
    Nov 19, 2012 at 14:16
8
$\begingroup$

Some month ago, I stumbled upon the highly recommended website of Hans-Martin v. Gaudecker who teaches courses like "Effective programming practices for economists". In his Autumn 2010 course he introduced SCons, in his Autumn 2011 course he switched to waf, which is supposed to be faster than SCons but still Python-based. The slides for both courses are available for download and I (as a social scientist) found them very instructive and enlightening.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ SCons is pretty rad. It supports very complicated or very simple schemae equally well! $\endgroup$
    – meawoppl
    Mar 15, 2012 at 21:00
  • 2
    $\begingroup$ The tradeoff between SCons and any faster build tool generally has to do with dependency checking. For mainstream languages (C, C++, Fortran, D, Python, Java, etc.), SCons will automatically determine dependencies using an MD5 hash-based algorithm, rather than time stamps, which can be fragile when dealing with generated files. Everything else beats SCons in performance (time needed to build software) because they don't do as much dependency checking, or they offload the dependency checking to some other tool (like the compilers used). $\endgroup$
    – Geoff Oxberry
    Mar 18, 2012 at 0:57
  • 1
    $\begingroup$ The first link of your answer is 404 now. It seems that his new page is at uni-bonn.de/~hmg308/teaching.html $\endgroup$
    – liori
    Aug 20, 2012 at 14:33
  • $\begingroup$ SCons has configurable "up-to-dateness" checking, so you can choose between timestamp, hash, or some combination. That said, I'm growing disenchanted with it: A few things are very easy (e.g. compiling software using a tool chain for which SCons has good Tool packages) and almost anything is possible, but it gets ugly pretty quickly. $\endgroup$
    – Eric Anderson
    Feb 24, 2014 at 17:13
4
$\begingroup$

Take a look at VisTrails. I haven't used it (only homebrew stuff around make), but it looks well thought-out, with good doc, and has real users at NASA etc.
(Are you looking for tools for 1-2 people, 4-5, more ?)

Added: not quite your question, but I think worth repeating:
for uniform, reproducible computer experiments one obviously needs

  • uniform directory structures, e.g. when-what/ in/ out/ scripts/ log/
  • uniform setting and echoing of all parameters for a run
  • scripts to summarize / plot / evaluate runs.

See also software-carpentry.org: "The problem we’re trying to solve is that scientists often spend 40% or more of their time wrestling with software, but 95% or more of them are primarily self-taught".

Improve this answer
$\endgroup$
4
$\begingroup$

All the requirements you mentioned in your question are fulfilled by the Swift parallel scripting system.

I've spent a year with Swift group as a postdoctoral researcher (PhD in scientific workflows). We've been helping scientists and researchers from different domains address their computational needs.

Swift is an open source framework for running workflows in parallel manner. It is called parallel scripting mainly to highlight the fact that it provides a scripting interface to creating workflows as opposed to the GUI box-arrow interface.

I can personally help you getting started and running your application with Swift. To know more about Swift, please take a look here.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Welcome to scicomp! Do you mind expanding your answer a little more (click the little gray edit button below your answer) to edit. Also, can you make your connection to Swift a little more clear in your answer? Thanks! $\endgroup$
    – Aron Ahmadia
    Nov 10, 2012 at 22:36
1
$\begingroup$

Taverna is an open-source WMS, not Python but Java.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Have you used it? $\endgroup$
    – Deathbreath
    Mar 13, 2012 at 20:53
  • $\begingroup$ Thanks for the suggestion. I saw the Taverna website, but it looks like a mainly graphical tool. I rather look for something command-line-based. Taverna does provide command-line tool, but it is only to execute workflows, but not build them (is it correct?). It also seems very much bioinformatics-oriented. $\endgroup$
    – btel
    Mar 14, 2012 at 10:31
  • $\begingroup$ It seems to me, you're more looking for a LIMS suitable for numerical experiments, rather than a build system like make or scons? $\endgroup$
    – GertVdE
    Mar 15, 2012 at 8:15
  • $\begingroup$ Sorry to ask. What does LIMS stand for exactly? $\endgroup$
    – btel
    Mar 15, 2012 at 14:38
  • 1
    $\begingroup$ Laboratory Information Management System. It's a family of tools to keep log of lab experiments. But these are typically, for example, for chemical analyses. You might want to Google for "in silico experiments", i.e. experiments that are simulations on a computer and require "logging" -> storing input/output data, what version of the software was used, hypotheses, ... $\endgroup$
    – GertVdE
    Mar 15, 2012 at 19:39
1
$\begingroup$

This page mentions some packages: https://wiki.python.org/moin/FlowBasedProgramming

Another promising package not mentioned here is Snakemake.

Improve this answer
$\endgroup$
0
$\begingroup$

Dexy sounds like it is exactly what you area after. From the site:

Dexy is a multi-purpose project automation tool with lots of features designed for working with documents. Dexy is written in Python and has a command-line interface. It's open source software with an MIT license.

What does Dexy do?

Dexy makes it easier to create technical documents by doing the repetitive parts for you. Dexy provides a consistent interface to tools and scripts so you don't have to run them manually. Your project's dexy configuration keeps track of what to run, in which order, and with what parameters. This way, your whole process is captured so anyone can run it using one simple command and the results will be consistent.

You want to write a blog post with examples showing how to use an API. Dexy will automatically:

  • run your example code, saving the results
  • apply syntax highlighting to your example code (using pygments)
  • insert the results of API calls and your prettified example code into your post (using jinja)
  • convert your markdown-formatted blog post to HTML (using python markdown or pandoc).
  • upload the HTML to the WordPress API in draft mode (using the WordPress API)
  • publish your blog post when you are finished tweaking it
Improve this answer
$\endgroup$
2
  • $\begingroup$ I've followed Dexy for a few years, and the impressions I've gotten are that it's not widely adopted, and it's not actively developed. These traits could be a chicken-and-egg problem (a small user base means it's not actively developed, not being actively developed hurts user base growth). It looks super cool, and on it's face, I think it's exactly what scientists need to broaden reproducibility beyond IPython, knitr, and bespoke scripts, but for some reason...it just doesn't seem like it gets used. Ana Nelson doesn't even blog that much about it, and she wrote it. $\endgroup$
    – Geoff Oxberry
    Aug 7, 2015 at 4:55
  • $\begingroup$ Well, the latest blob post is from January, and there have been 3 commits this year. Not super active, but not dead, especially if it's one of those project that just gets to stable and doesn't really need any more work. There are other project I use with much deader recent development histories. As for the chicken-and-egg problem, maybe an upvote here and wherever else it's mentioned on SE would help :P $\endgroup$
    – naught101
    Aug 11, 2015 at 5:30
0
$\begingroup$

Knime or Rapidminer could likely handle all the requirements as well. Whether they should is up for grabs :)

Improve this answer
$\endgroup$
2
  • $\begingroup$ Maybe Kepler too. $\endgroup$
    – Grimbly
    Mar 16, 2016 at 23:52
  • $\begingroup$ Also, Nifi is quite nifty, but I don't think it fully covers all of the requirements. $\endgroup$
    – Grimbly
    Mar 17, 2016 at 0:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.