What guidelines should I follow for simulation software projects?

Question

I am not sure whether this question belongs here, but I would like to give it a try and benefit from the experience of the people at scicomp.SE.

From my experience, the software quality in computational science often leaves a bit to be desired. Clearly, established software projects such as PETSc or OpenFOAM are exceptions to that statement. But often there is some old self-crafted code from the 80s that should serve as basis for new research topics but no one really understands what it does. I have the impression that while there are trends and guidelines to make simulation software projects more sustainable, often the developers lack the skills to implement quality assurance measures. Not blaming anyone, this is an interdisciplinary field and in a physics or engineering degree those skills are not taught.

1. Are there some established guidelines that one can follow?

Thanks to contributors from other channels and my own research I can provide this list:

• Guidelines by the Netherlands eScience Center
• Guidelines by the DLR

Are there further candidates?

2. Are there any template projects that provide a clean basis?

In order to make things easier for developers of simulation software projects, I thought it would be helpful to have some boilerplate project that provides a clean start. Is there already such a thing?

So far I have collected several components the boilerplate project should contain:

• Version control system (git)
• Collaboration tool (gitlab)
• Continuous integration (gitlab-ci) for automatic build tests
• Extract documentation from source code (doxygen)
• Automated build system (CMake)
• Framework for unit tests (Google Test? Catch2)
• Code formatting tool (clang-format)
• ...

but I fear that the list is not exhaustive and would be grateful for your experiences and/or pointers to literature.

EDIT: The second question has lead to a project: Check out bertha, the skeleton [1, 2]! The goal is to install this project as template in GitLab so that with a click an instance of a working C++ project that follows the best practices for simulation software projects is generated. Of course, it is also possible to copy the files manually. Anyway, instead of starting from scratch the skeleton project provides a solid base. Alternatively, one can cherry-pick things from it if there is already an established software project.

At the moment, bertha features an automated multi-platform build system, automatic documentation generation, and supports unit tests using the Catch2 framework. It is about to be extended, so if you have any input do not hesitate to contact me or raise an issue on GitLab.

[1] https://gitlab.com/cph-tum/bertha

[2] https://arxiv.org/abs/1912.01640

Answer 1

"developers lack the skills".

Maybe.

I think it's much more likely that the developers lack the incentives. Making solid code is difficult and expensive and, in academia, comes with minimal-to-negative reward. You're asking for a list of things of guidelines, but all of your examples are specific to the technical situation, not the social situation. That's asking for trouble.

One way to get good software is to change the incentives. In my work as an editor, I send papers back if the authors haven't released their source code or that source code doesn't meet my (admittedly self-defined) standards. Some journals, such as JOSS, take this farther and have guidelines for what they expect to see. If you find yourself in a position of power as a reviewer or editor, use that influence to help move your field into the 21st century.

If you're a student, or mentor students, you should know that it's hard to get tenure. A reasonable person will therefore seek to develop diversified skills during their PhD. They say that Github is the new resume. Having solid, unit-tested, documented code is a valuable indicator for alternative academic tracks (research programmer) as well as for industry and government. Use this as a carrot for yourself and others.

As the JOSS guidelines say, you should have a contributor guidelines for your project and maybe a PR template. If you want good code, you need to make it easy for people to help you build it. If you're in a senior position, you also need a way of educating your mentees and, especially, yourself. Programs like Software Carpentry can help with this.

In short, with some remarkable exceptions, software is only as good as the incentives which produce it.

I also highly recommend the paper "Good enough practices in scientific computing".

Answer 2

I maintain (and am the main coder of) a simulation software that has been developed for ~8 years and is used by few hundreds people. It all started as a side project during my PhD, and it clearly outgrew itself. It is both over- and under-engineered: the architecture of some parts is too complicated for their own good, whereas some other parts (whose importance increased over time) were not designed carefully enough at the beginning. I'll try to summarise my experience in a few points/guidelines I would have loved to be told when all started:

• As soon as your code is usable (that is, it can be used in a research project) start working on usage examples and a user manual. Keeping them updated will be a lot of work but it will make your life much easier later on.
• About the in-source documentation: keeping the doxygen stuff up to date will require much more work than one would naively expect. Be sure to have a developer team that is keen on keeping it updated.
• Ask for user feedback from the start and be ready to tune the architecture as you increase the features.
• On a similar point, be ready to refactor the code quite often, at least during the early stages. I think the risk of over-engineering at the beginning is very high. If you are not 100% sure of what you need, it is better to start with a lightweight code and refactor it later than start with a behemoth-like architecture which will be very hard to touch later on. This is an issue I am fighting against right now: a decision I made at the very beginning is really hurting my ability of adding a feature I dearly need.
• Be careful when adding dependencies: successful simulation codes last for many years, and small libraries you may be tempted to include to avoid having to write bits of code might not be as long-lived. In addition, simulation codes are usually geared towards performance: if you want to include libraries in your kernels, be sure that their performance are up to the task.

Answer 3

tl;dr: Look outside our field, start by following the Linux Foundation Core Infrastructure Initiative Badge guidelines and then maybe also have a look at the xSDK policies for HPC simulation software.

---

This question is really good, as it depicts the panic of every research software developer that discovers new possibilities for improving the non-functional aspects of their software.

I think that we should avoid just copying practices from other scientific computing projects. Yes, you will definitely find good practices in several (very important and successful) projects, but our field is often an echo chamber of ideas that one colleague introduced in the past, they worked for a while, but nobody outside our field evaluated. And yes, our projects do have some special needs, but for many of the problems, current mainstream solutions are already good enough.

Example case: OpenFOAM is maybe the most used free/open-source simulation framework out there at the moment. But I literally just found this question while trying to find if anyone out there has proposed automatic code formatting for OpenFOAM. In that community, "clean code" still includes "it follows the style guide", which needs to be done manually. A few minutes before, I was trying to make my IDE find the headers, because OpenFOAM does not use a common build configuration system (for historical and backwards-compatibility reasons).

Following the lists of guidelines cited in the original question and in the other answers is already a very good step, keeping in mind that some of these may be looking into specific fields or use cases.

If you are looking for an overview of more general best practices, followed by a few thousand projects from different fields (not only research software), you can follow the Linux Foundation (LF) Core Infrastructure Initiative (CII) Best Practices badge.

For simulation software, you may want to also look at the xSDK policies for more technical guidelines on scientific software and specifically for high-performance computing.

Getting the "passing" level should be easy for most healthy projects of any size nowadays. Going to higher levels would probably make sense for projects with several developers and users, which is often not the case with research software.