# What are good ways to document scientific software?

Many times, when I've inherited or encountered scientific code written by other people (or occasionally, even my own work), I've noticed that documentation is either sparse or nonexistent. If I'm lucky, I see informative comments. If I'm very lucky, there's even Doxygen comments and a Doxyfile so that I have function interfaces and some formatted HTML to consult. If I'm extremely lucky, there's a PDF manual and examples in addition to the Doxygen and source file comments, and I'm ecstatic, because it makes my life much, much easier.

What information and tools are useful in documenting source code? For that matter, what information and tools are useful to document the data and results that accompany that source code, in the case of scientific software?

• A related programmers.SE thread. – J. M. Dec 10 '11 at 12:25
• In R, one could use roxygen(2) and/or Sweave to document code and write vignettes (manuals). – Roman Luštrik Dec 10 '11 at 15:17
• An excellent example are the deal.ii tutorials which not only teach you how to use the software but also how finite elements work. – David Ketcheson Dec 10 '11 at 16:10
• I was recommended M2HTML to make documentation of matlab code easier. – Artem Kaznatcheev Jan 10 '12 at 3:06
• org-mode is brilliant multi-lingual literate programming – David LeBauer Mar 17 '12 at 4:02

I think that documentation for scientific software can be divided into three categories, all of which are necessary for full understanding. The easiest and most common is individual method documentation. There are many systems for this. You mention doxygen, Python has pydoc, and in PETSc we have our own package sowing which generates the following.

However, for any piece of software which goes beyond a simple utility, you need a manual. This provides a high-level view of the purpose of the package, and how its different functionalities integrate to achieve this purpose. It helps a new user structure their code, often through the use of examples. In PETSc, we just use LaTeX for the manual, but the PyClaw package uses the Sphinx framework which I am very impressed with. One thing that we have implemented in the sowing package that I find very useful is the link between example code and function documentation. For example, this example solves the Bratu equation. Notice how you can follow the links for any custom type or function call and get to the low-level documentation, and how those pages link back to examples using them. This is how I learn about new functionality which other people in the project contribute.

A frequently overlooked part of documentation, I think, is developer documentation. It is not uncommon to publish a coding-style document, and instructions for interacting with the repository. However, it is very rare to explain the design decisions made before implementation. These decisions always involve tradeoffs, and the situation with respect to hardware and algorithms will necessarily change over time. Without a discussion of the tradeoffs reviewed and rationale for particular design decisions, later programmers are left to recreate the entire process on their own. I think this is a major impediment to successful maintenance and improvement of old codes when the original developers are no longer in charge.

• +1 for Sphinx! Note that it includes autodoc, which I think is much superior to pydoc. – David Ketcheson Dec 10 '11 at 16:14
• +1 for the separation into interface/user/developer documentation. – Florian Brucker May 23 '13 at 9:38

### In-code documentation

The most important thing is to use the documentation facilities in your chosen development environment, so that means pydoc for python, javadoc in java or xml comments in C#. These make it easy to write the documentation at the same time as writing the code.

If you rely on coming back and documenting things later, you may not get around to it, but if you do it as you are writing the code, then what needs to be documented will be fresh in your mind. C# even has the option to issue a compilation warning if the XML documentation is incomplete or inconsistent with the actual code.

### Tests as documentation

Another important aspect is having good integration and unit tests.

Often documentation concentrates on what classes and methods do in isolation, skipping over how they are used together to solve your problem. Tests often put these into context by showing how they interact with each other.

Similarly, unit-tests often point out external dependencies explicitly through which things need to be Mocked out.

I also find that using Test-driven development I write software which is easier to use, because I'm using it right from the word go. With a good testing framework, making code easier to test and making it easy to use are often the same thing.

### Higher level documentation

Finally there is what to do about system level and architectural documentation. Many would advocate writing such documentation in a wiki or using Word or other word processor, but for me the best place for such documentation is alongside the code, in a plain text format that is version control system friendly.

Just like with in-code documentation, if you store your higher level documentation in your code repository then you are more likely to keep it up to date. You also get the benefit that when you pull out version X.Y of the code, you also get version X.Y of the documentation. In addition, if you use a VCS friendly format, then it means that it is easy to branch, diff and merge your documentation, just like your code.

I quite like reStructuredText (rst), as it is easy to produce both html pages and pdf documents from it using sphinx, and is much friendlier than LaTeX, yet can still include LaTeX math expressions when you need them.

• I would like to point to Lyx (lyx.org) for writing LaTeX documents for supporting documentation for code. – ja72 Feb 28 '12 at 18:42
• I've used Lyx in the past, but the thing I like about rst is that I can write it in a normal text editor (in the same IDE I use to write code) and still be pretty sure I know what the final document will look like. Also the formatting conventions make it very VCS friendly, which is something that's important to me. – Mark Booth Feb 28 '12 at 18:53

I will take objection with almost every point Faheem makes. Specifically:

1/ "I think that it is unrealistic to expect scientific developers to spend a great deal of time documenting their software". This is a prescription for a failed project that soon nobody will be able to use who does not have access to the primary developers. It is for good reason that the big scientific computing libraries (e.g. PETSc, or deal.II) have extensive documentation that runs into the 1,000s of pages or more. You can't have a sizable user community if you don't have extensive documentation. I will agree, however, that example codes need to be simple and focused on a single concept.

2/ "the authors should consider writing a paper for publication if appropriate". That's often not possible in practice. One can write papers on concepts and algorithms, but not on interface and other design decisions. Readers of such papers will get an idea what the implementation does it; users of the implementation will need to know what functions to call, what the arguments means, etc. As a user, one can most of the time live without the former and simply consider a library as a black box, but one can not do without interface information.

• Welcome, Wolfgang; I think you are the right person to answer this question, but I have a suggestion: what you've written here should perhaps be a comment on Faheem's answer, rather than an answer to the question itself. – David Ketcheson Dec 16 '11 at 8:30
• I see now, indeed. I think I hadn't realized at the time how this works. – Wolfgang Bangerth Dec 16 '11 at 14:52
• @WolfgangBangerth: Thanks for your comments, which I didn't see because I was not notified. I think that maybe an @ in front of the Faheem would have done it, but I don't have a good reference. I'll attempt to address your comments in my answer - there is not enough space in a comment. – Faheem Mitha Jan 10 '12 at 4:34
• @FaheemMitha: Did you write the answer? The problem with new answers to a question is that they are re-ordered based on how many up-/downvotes they get whereas comments remain linear... – Wolfgang Bangerth Jan 10 '12 at 13:15

This is a good question. To a first approximation, the code should attempt to be self documenting. So, for example, if the software is command line, you should be able to do executable --help or executable -h or even executable (if the executable does nothing with no arguments), and have a brief usage message return.

Second, I think that it is unrealistic to expect scientific developers to spend a great deal of time documenting their software, so I suggest keeping it simple. A short text manual with the basic methods and options and annotated working and tested examples of usage, graduated from simple to more complex (usage examples are very important and frequently neglected) is considerably better than nothing and much more than most scientific software offers. I'd also like to add a pet peeve about usage examples. Simple means simple. That means if you are trying to illustrate a method, you don't add in ten extraneous concepts or methods to confuse the reader. Keep it simple and annotate so the reader knows what the example is supposed to be doing. I'd also suggest tying the manual usage examples into a test suite somehow so they continue working when the code is changed. I don't actually know how to do this in a nice way, so please feel free to educate me. If the developers want to get more fancy, sure they can use nice markup languages and so forth, add man pages and so on. If the software is mathematical, plain text will of course not work.

Third, the authors should consider writing a paper for publication if appropriate. This would usually address design decisions and give a more high level perspective on the software than a manual does, or can be expected to do. This would address the documentation of design decisions that @Matt was talking about.

Of course, the most important documentation of all is the code itself, which should be commented as necessary. This assumes the code is free software. If it isn't, then it is substantially less useful as scientific software (do you really want to use a black box where you can't see how the methods are implemented?) and I for one would not use it.

To address your question on how to document data and results, I'd recommend something like Python's doctest module. This allows you to write tutorials or tests in a way that can be automatically validated.

If you are interested in literate programming, have a look at org-babel. It is part of org-mode in Emacs and thus gives you a wide array of export options (LaTeX,PDF,HTML,ODT) for documentation. Emacs can display images inside the buffer and let you write mathematical equations in LaTeX syntax so you don't have to limit yourself to plain text documentation.

• A relevant feature of org-mode is that it executes c, SQL, bash, R, python, and many other languages, seamlessly in the text. – David LeBauer Mar 17 '12 at 4:16

Documentation that is automatically derived from your source code is an essential component in having up-to-date i.e. correct documentation. I can't count how many times I've seen documentation that is years behind the source code because of developer apathy towards documentation. The easy solution is to make it easy for programmers to write documentation along with new code in the same place at the same time rather than as some a posteriori effort that will inevitably be de-prioritized in favor of more glorious activities.

If forced to choose, I'd rather have detailed and accurate (i.e. current) source code comments but no user manual whatsoever than an out-of-date user manual that's full of wrong information.

In Python there are the tools pep8 and pep257 which will report missing or malformed documentation. elpy for Emacs will also complain about missing documentation. The Numpy docstring conventions with reStructuredText is good to follow. Test with pep8, pep257 and doctest can be setup with py.test and tox running automatically.