19

A lot has been written about how to design, execute, and report the results of computational experiments. This has obvious connections with open source software and the broader "open science" movement. Another important issue is the difference between "my code is faster than yours" research and research that helps us to better understand the properties of ...


14

To make more robust comparisons (on linux), you can : 1) On Intel CPUs the turbo overclocks your CPU. This is controlled by the temperature of the CPU, so it can behave differently from one run to the other. On Linux, you can block the frequency of the CPU as follows. For example, for 2.4GHz: echo 1 > /sys/module/processor/parameters/ignore_ppc for ...


9

I'm a happy user of GoogleTest with a C++ MPI code in a CMake/CTest build environment: CMake automatically installs/links googletest from svn! adding tests is a one-liner! writing the tests is easy! (and google mock is very powerful!) CTest can pass command-line parameters to your tests, and exports data to CDash! This is how it works. A batch of unit-...


8

the planned longevity of TeX comes to mind: “Ever since those beginnings in 1977, the TeX research project that I embarked on was driven by two major goals. The first goal was quality: we wanted to produce documents that were not just nice, but actually the best. (…) The second major goal was archival: to create systems that would be independent of ...


7

You might consider writing the entire paper in Noweb. It's a bit tedious to set up, but it's a very powerful way to mix code and LaTeX-formatted text, equations, and figures. For long programs, it tends to turn your code into more of a book than an article, but for short programs, it might work out pretty well. If you don't want to go that far, it still ...


6

Any numerical differences between A and B will become exponentially larger with time (i.e. the Lyapunov instability, as discussed in Frenkel and Smit). Even a small difference due to basis set size could result in dramatic differences in the trajectory over time. So I'm not sure that a comparison between individual trajectories will be meaningful. It may be ...


6

In deal.II, we have a testsuite that is driven by a regular Unix Makefile. It has a default target that runs all the usual tests, and a separate target for expensive tests. Running each test is done using a generic rule but the default target calls the generic rule only for certain tests and the expensive target calls it for the expensive tests. Because it's ...


6

For the most part I ditto Jed's answer. However, there is a different way out: Given the size of normal floating point numbers, you can store every number in a 4000-or-so bit fixed point number. So if you do a reduction on floating point numbers thus embedded, you get an exact calculation, no matter the associativity. (Sorry, I don't have the reference to ...


5

I would've posted this in a comment, but I unfortunately don't have enough reputation. This answer (https://stackoverflow.com/a/42610074/9796552) should provide enough to help you achieve what you want. In short, you can use Julia's Sys module which contains functions dedicated to retrieving system information.


5

The noweb approach mentioned by Bill has evolved quite a bit, both in it's original spirit of documenting code (rather than scientific publication) under the term literate programming and now comes in many flavors (I guess noweb was a generalization of cweb initially), of which doxygen and various language specific versions can generate documentation in TeX, ...


4

As Paul mentioned in his comment, git and public hosting sites like Github and Gitlab are invaluable for keeping track of the development history of computational projects. A few concrete examples of where I've found that history useful: US federal grants require you to fill out yearly progress reports and I can never remember exactly what I did, but I can ...


4

There are more possibilities, e.g. Sharing a Docker image containing the full conda environment, including all packages and associated code Sharing a virtual machine image containing either the Docker image from 3 or directly including the conda environment from 1. Options 3 and 4 require only the ability to run Docker or VM images and no longer depend on ...


4

There are many technical challenges that make exact bit-for-bit reproducibility of computational results extremely hard to achieve. At the software level, changes to the code or any of the libraries used by the code can obviously cause different results to be produced. You'd be surprised by the number of support libraries that can end up linked into a ...


4

There are several MPI-enabled software packages that use the CMake set of tools for testing. The ones that I can think of off the top of my head are Trilinos, VTK and ParaView. I would think that you don't want to assume that the executable needs to be launched with mpirun and/or mpiexec. CMake has support for specifying how to properly launch the executable ...


4

We simply roll our own code in deal.II -- in essence, we tell the framework to execute tests using mpirun -np .... We had previously just used a Makefile-based testing scheme (compile, link, execute test, then compare the output with one that had previously been saved) and you can find this here: https://svn.dealii.org/branches/releases/Branch-8-0/tests/mpi/...


3

The Teuchos Unit test harness in Trilinos natively supports unit tests that use MPI. Things like controlling output from multiple processes and aggregating pass/fail over all processes is automatic. Take a look: http://trilinos.org/docs/dev/packages/teuchos/doc/html/group__Teuchos__UnitTest__grp.html


3

The LaTeX package minted provides very extensive syntax highlighting (based on Pygments) and allows cross-referencing in both directions. You can escape to LaTeX from within the code part (the minted part) and you can refer in your main text to lines of code. On top of that, it provides a listings environment so that you can generate a "list of listings" (...


2

First thought is to use A and B on standard validation test cases. I'm not sure what's available for validation on MD (googling "molecular dynamics validation" turned up a lot), but in CFD there's plenty of databases. If you need to validate something very specific (it sounds like you are), then you'll need to compile some statistics to convince me that A ...


2

There have been many attempts at making reproducibility happen and there is a whole literature on this topic. My personal opinion from 15 years of scientific software is that it's unrealistic, as unsatisfactory as I find that answer. The problems are that (i) complex software has bugs and so can't be frozen; (ii) software is never feature complete and so ...


2

I would like to point out that instead of using higher precision arithmetic for the addition, there's the possibility of using compensated summation (see [1]). This could increase the accuracy of the summation without the need to resort to larger data types. [1] Higham, N. J. The Accuracy of Floating Point Summation. SIAM Journal on Scientific Computing 14,...


2

For a possible solution to this problem, see my ActivePapers project. In summary, it describes how data and code can be packaged together with explicit dependencies on specific versions of each software component. This makes it possible to exactly reproduce a computation, while also permitting to run updated software on the same data. I should add that ...


1

I believe that as far as the choice of language goes, using a standardized one (e.g. C/Fortran/C++) would qualify as "best practice". If a package depends on 10 other libs/packages, especially those written in obscure languages then that obviously is bad for longevity. Many projects end up being orphaned after some time. I dont think major libs/api's like ...


Only top voted, non community-wiki answers of a minimum length are eligible