Citable references for software best practices

Question

I'm currently writing up my PhD thesis. I spent a significant fraction of my PhD cleaning up and extending existing scientific code, applying software engineering best practices which were previously not used, and would like to write about this in my thesis. Rather than simply saying "I added unit tests", I want to be able to write something like this:

J. Doe invented unit tests in 1975$^{[23]}$. A recent study by Bloggs et al$^{[24]}$ showed that unit tests reduce the incidence of software errors by 73%... 234 separate unit tests were added to the code base, managed by the xUnit framework created by Timpkins et al$^{[25]}$

I'm looking for citable academic references (preferably articles in peer-reviewed journals where I can get DOIs, BibTeX etc) to widely accepted software engineering best practices, specifically:

• unit tests
• version control
• modularisation / separation of concerns
• performance profiling / optimisation based on profiling information
• bug/issue tracking

I'm looking for information both about the initial invention and on subsequent evaluations of effectiveness. If there's a review article that lists all of this stuff in one place then so much the better.

Answer 1

Steve McConnell's book Code Complete, 2nd edition has an extensive bibliography discussing these issues from more of the standpoint of software developers than computational scientists. The book is starting to become a little dated, in that it's approaching a decade old, so it doesn't cover more recent testing methodologies like behavior-driven development. Nevertheless, it is the closest thing to a comprehensive review article on software construction that I am aware of. You could also look for articles in IEEE Software.

On the computational science side of things, I think the best article is probably the PLoS version of the arXiv preprint DavidKetcheson cited on "Best Practices for Scientific Computing". I say this coming from an engineering background, where fewer people cite arXiv references or post arXiv preprints, and thus, citing a "real journal article" (setting aside, of course, all of the issues about scientific publishing that are being debated right now) is looked upon more favorably (and I get the impression that is why those authors chose to publish it in a journal).

The authors of the PLoS paper that DavidKetcheson and I cited are part of an organization called Software Carpentry that puts on (usually 2 day) "boot camps" to teach scientists about some best practices for software development and useful computational skills for scientists (not just computational scientists). The Software Carpentry web site has an extensive bibliography related to software development in science. If you're interested in these issues, I encourage you to reach out to them; they're always looking for more advocates of best practices in software development to do volunteering in various capacities. (Disclaimer: I volunteer with Software Carpentry.)

Another common justification for engaging in software development best practices is reproducibility. Victoria Stodden has curated a long list of reproducible research references that may be of interest, depending on what you want to say.

Answer 2

I don't have references for the origin of each of these ideas/practices. But here are some very recent, relevant references:

• What makes computational open source software libraries successful?
• Best Practices for Scientific Computing

Answer 3

IMHO I would take great care citing "Best Practices" in the context of scientifically proven approaches. Most practices are derived from "what seems to work for a set of projects by someone perceived as guru involved in those projects", rather than derived from rigorous testing of different approaches. There are just too many variables and human factors in software engineering to state there is a referable list of "best practices" (e.g. a practice that works on one project will completely fail on another).

I would approach it by stating what your project needed, why it needs it and add references to methods used and why you used them.

I would also lean towards reporting quantifiable results rather than references to state your point. For example, if your unit tests uncovered 100 bugs, 10 of which serious enough to cast doubt on previously published result. This is a much more powerful statement to have in your PhD than a statement that you know the origin of unit tests.

edit: (fixed typo) - Answering the following - I often give raising children as an analogy to software projects. There there are many methods and tested ways to raise them, trying to raise your children with one method because it works for the average or a tested subsample, will work as long as your child is the same as the one tested. It is better to know many methods and apply the ones that work in your instance. Yes unit testing may be proven, but applying it based on that alone could mean your project gets to market late and therefore fails it's objective (if that is the objective). I am saying that applying a method to get an outcome and giving the result of that outcome, in my opinion is better in a thesis, than listing things that you tried based on other projects - unless of course the topic of the thesis is measuring methodologies :)