# Citable references for software best practices

I'm currently writing up my PhD thesis. I spent a significant fraction of my PhD cleaning up and extending existing scientific code, applying software engineering best practices which were previously not used, and would like to write about this in my thesis. Rather than simply saying "I added unit tests", I want to be able to write something like this:

J. Doe invented unit tests in 1975$^{[23]}$. A recent study by Bloggs et al$^{[24]}$ showed that unit tests reduce the incidence of software errors by 73%... 234 separate unit tests were added to the code base, managed by the xUnit framework created by Timpkins et al$^{[25]}$

I'm looking for citable academic references (preferably articles in peer-reviewed journals where I can get DOIs, BibTeX etc) to widely accepted software engineering best practices, specifically:

• unit tests
• version control
• modularisation / separation of concerns
• performance profiling / optimisation based on profiling information
• bug/issue tracking

I'm looking for information both about the initial invention and on subsequent evaluations of effectiveness. If there's a review article that lists all of this stuff in one place then so much the better.

• Does this help: plosbiology.org/article/… – akid Jan 17 '14 at 13:53
• If the purpose of adding references is to convince readers that better practices are better, it might make more sense to explain why they are better directly; simply giving references might be less persuasive. Bear in mind that a lot of these things are common in undergraduate software engineering courses, can be found in standard textbooks, and are not necessarily cutting edge research. – Kirill Jan 17 '14 at 15:19
• My experience is that you need both motivation and references. I just had a conversation with co-workers yesterday (both of whom are practicing scientists) who were of the opinion that ad hoc testing methodologies work better (short answer: they don't). It is important to express the motivation in terms of metrics that computational scientists seem to care about: more higher impact papers faster, and more correct results (see the link about reproducible research). Point to references because people will fight you on these points claiming that there are no significant benefits. – Geoff Oxberry Jan 18 '14 at 3:17
• In all likelyhood the people who will be examining my thesis will be chemistry or materials science professors rather than computational science experts. They will probably have some experience writing code but they will almost certainly not have done any serious coding since they were students or early post-docs themselves. What I need is something that says "That year of my PhD that I spent on this, I was actually doing something useful and not just slacking off" – user1915639 Jan 21 '14 at 12:59

Steve McConnell's book Code Complete, 2nd edition has an extensive bibliography discussing these issues from more of the standpoint of software developers than computational scientists. The book is starting to become a little dated, in that it's approaching a decade old, so it doesn't cover more recent testing methodologies like behavior-driven development. Nevertheless, it is the closest thing to a comprehensive review article on software construction that I am aware of. You could also look for articles in IEEE Software.

On the computational science side of things, I think the best article is probably the PLoS version of the arXiv preprint DavidKetcheson cited on "Best Practices for Scientific Computing". I say this coming from an engineering background, where fewer people cite arXiv references or post arXiv preprints, and thus, citing a "real journal article" (setting aside, of course, all of the issues about scientific publishing that are being debated right now) is looked upon more favorably (and I get the impression that is why those authors chose to publish it in a journal).

The authors of the PLoS paper that DavidKetcheson and I cited are part of an organization called Software Carpentry that puts on (usually 2 day) "boot camps" to teach scientists about some best practices for software development and useful computational skills for scientists (not just computational scientists). The Software Carpentry web site has an extensive bibliography related to software development in science. If you're interested in these issues, I encourage you to reach out to them; they're always looking for more advocates of best practices in software development to do volunteering in various capacities. (Disclaimer: I volunteer with Software Carpentry.)

Another common justification for engaging in software development best practices is reproducibility. Victoria Stodden has curated a long list of reproducible research references that may be of interest, depending on what you want to say.

• The "Software Carpentry" reading list looks helpful. – user1915639 Jan 21 '14 at 13:06

I don't have references for the origin of each of these ideas/practices. But here are some very recent, relevant references:

• I definitely second the first of these references :-) It's complete information is Wolfgang Bangerth, Timo Heister What makes computational open source software libraries successful? Computational Science & Discovery, vol. 6, article 015010 (18 pages), 2013 – Wolfgang Bangerth Jan 18 '14 at 1:24

IMHO I would take great care citing "Best Practices" in the context of scientifically proven approaches. Most practices are derived from "what seems to work for a set of projects by someone perceived as guru involved in those projects", rather than derived from rigorous testing of different approaches. There are just too many variables and human factors in software engineering to state there is a referable list of "best practices" (e.g. a practice that works on one project will completely fail on another).

I would approach it by stating what your project needed, why it needs it and add references to methods used and why you used them.

I would also lean towards reporting quantifiable results rather than references to state your point. For example, if your unit tests uncovered 100 bugs, 10 of which serious enough to cast doubt on previously published result. This is a much more powerful statement to have in your PhD than a statement that you know the origin of unit tests.

edit: (fixed typo) - Answering the following - I often give raising children as an analogy to software projects. There there are many methods and tested ways to raise them, trying to raise your children with one method because it works for the average or a tested subsample, will work as long as your child is the same as the one tested. It is better to know many methods and apply the ones that work in your instance. Yes unit testing may be proven, but applying it based on that alone could mean your project gets to market late and therefore fails it's objective (if that is the objective). I am saying that applying a method to get an outcome and giving the result of that outcome, in my opinion is better in a thesis, than listing things that you tried based on other projects - unless of course the topic of the thesis is measuring methodologies :)

• Studies have, in fact, compared defect detection strategies such as unit testing, pair programming, stepping through a program with a debugger, and formal code review and rated their efficacy. You're right that each strategy has its place. The software development community recognizes this point in the literature, and suggests what might work best for different types of projects. If "too many variables and human factors" were really an obstacle to formulating best practices, we wouldn't have them in medicine, or other fields with similar complex issues, yet we do. I don't buy your argument. – Geoff Oxberry Jan 18 '14 at 10:22
• "a mush more powerful statement in your PhD" is a lovely typo – denis Feb 10 '14 at 13:48