20

What you are asking for is the Elsivier grand challenge of the "Executable Paper". While many approaches have been tried, none are as compelling as the authors might suggest. Here are a few examples of techniques used. Madagascar Project takes your approach, inside the make script have the simulations run that produce the figures and paper simultaneously. ...


19

A lot has been written about how to design, execute, and report the results of computational experiments. This has obvious connections with open source software and the broader "open science" movement. Another important issue is the difference between "my code is faster than yours" research and research that helps us to better understand the properties of ...


15

A reduction implemented using MPI_Allreduce() is reproducible as long as you use the same number of processors, provided the implementation observed the following note appearing in Section 5.9.1 of the MPI-2.2 standard. Advice to implementors. It is strongly recommended that MPI_REDUCE be implemented so that the same result be obtained whenever the ...


14

As some comments have suggested, this approach has long been developed in the R community by building on Sweave and more recently, knitr. Obviously this approach has the disadvantage of being language-specific at the moment, but the advantage that its regularly used in academic papers. Use of Sweave in real publications The Journal of Biostatistics ...


14

To make more robust comparisons (on linux), you can : 1) On Intel CPUs the turbo overclocks your CPU. This is controlled by the temperature of the CPU, so it can behave differently from one run to the other. On Linux, you can block the frequency of the CPU as follows. For example, for 2.4GHz: echo 1 > /sys/module/processor/parameters/ignore_ppc for ...


13

I have not had a lot of success in using other people's solutions to this problem. I usually just want something simple that works for me and gets the job done. To this end, I generally try to write one python script which is in charge of running all the results, parsing the output, as well as building the figures/tables. I write my codes to generate ...


9

I'm a happy user of GoogleTest with a C++ MPI code in a CMake/CTest build environment: CMake automatically installs/links googletest from svn! adding tests is a one-liner! writing the tests is easy! (and google mock is very powerful!) CTest can pass command-line parameters to your tests, and exports data to CDash! This is how it works. A batch of unit-...


8

the planned longevity of TeX comes to mind: “Ever since those beginnings in 1977, the TeX research project that I embarked on was driven by two major goals. The first goal was quality: we wanted to produce documents that were not just nice, but actually the best. (…) The second major goal was archival: to create systems that would be independent of ...


7

You might consider writing the entire paper in Noweb. It's a bit tedious to set up, but it's a very powerful way to mix code and LaTeX-formatted text, equations, and figures. For long programs, it tends to turn your code into more of a book than an article, but for short programs, it might work out pretty well. If you don't want to go that far, it still ...


7

Even more important, in my opinion, is making sure that you can figure out how to re-generate all your results from scratch in a month or a year (for instance, when referees ask you to add or modify something). For that purpose, what I do is include a text file with very detailed directions on how to reproduce all results. It's best if you test these out ...


6

Emacs's orgmode in combination with Babel achieves that. Babel can execute code snippets from various programming and scripting languages, e.g., it could open the file containing the simulation data and put it into a table in orgmode, which can be exported to LaTeX (and many other formats). It takes quite a while getting used to all the key combos in orgmode,...


6

For the most part I ditto Jed's answer. However, there is a different way out: Given the size of normal floating point numbers, you can store every number in a 4000-or-so bit fixed point number. So if you do a reduction on floating point numbers thus embedded, you get an exact calculation, no matter the associativity. (Sorry, I don't have the reference to ...


6

Any numerical differences between A and B will become exponentially larger with time (i.e. the Lyapunov instability, as discussed in Frenkel and Smit). Even a small difference due to basis set size could result in dramatic differences in the trajectory over time. So I'm not sure that a comparison between individual trajectories will be meaningful. It may be ...


6

In deal.II, we have a testsuite that is driven by a regular Unix Makefile. It has a default target that runs all the usual tests, and a separate target for expensive tests. Running each test is done using a generic rule but the default target calls the generic rule only for certain tests and the expensive target calls it for the expensive tests. Because it's ...


5

The noweb approach mentioned by Bill has evolved quite a bit, both in it's original spirit of documenting code (rather than scientific publication) under the term literate programming and now comes in many flavors (I guess noweb was a generalization of cweb initially), of which doxygen and various language specific versions can generate documentation in TeX, ...


5

You can implement a numerically stable reduction algorithm in MPI the same as you can do in serial. There may be a performance hit, of course. If you can afford to replicate the vector, just use MPI_Gather and do the numerically stable reduction in serial on the root. In some cases, you may find the performance hit is not a big deal. Another solution is ...


5

I would've posted this in a comment, but I unfortunately don't have enough reputation. This answer (https://stackoverflow.com/a/42610074/9796552) should provide enough to help you achieve what you want. In short, you can use Julia's Sys module which contains functions dedicated to retrieving system information.


4

If running all your code is cheap then you could do something low-tech like the following: You could template your documents with formatted strings so that they look like this "we observed a %(fractional_improvement)s increase in ..." Have python scripts that look like this results = {"alpha" : run_alpha_computation(...), "...


4

If you're using LaTeX, a relatively low-tech solution is to have your code spit out a file (or else use a script to filter from your code's output) containing a whole bunch of lines like this: \newcommand{\myresults1}{<value>} Then you can use the \input command to add that file into your document, and use the defined commands to place the values.


4

There are several MPI-enabled software packages that use the CMake set of tools for testing. The ones that I can think of off the top of my head are Trilinos, VTK and ParaView. I would think that you don't want to assume that the executable needs to be launched with mpirun and/or mpiexec. CMake has support for specifying how to properly launch the executable ...


4

We simply roll our own code in deal.II -- in essence, we tell the framework to execute tests using mpirun -np .... We had previously just used a Makefile-based testing scheme (compile, link, execute test, then compare the output with one that had previously been saved) and you can find this here: https://svn.dealii.org/branches/releases/Branch-8-0/tests/mpi/...


4

There are many technical challenges that make exact bit-for-bit reproducibility of computational results extremely hard to achieve. At the software level, changes to the code or any of the libraries used by the code can obviously cause different results to be produced. You'd be surprised by the number of support libraries that can end up linked into a ...


4

There are more possibilities, e.g. Sharing a Docker image containing the full conda environment, including all packages and associated code Sharing a virtual machine image containing either the Docker image from 3 or directly including the conda environment from 1. Options 3 and 4 require only the ability to run Docker or VM images and no longer depend on ...


3

The LaTeX package minted provides very extensive syntax highlighting (based on Pygments) and allows cross-referencing in both directions. You can escape to LaTeX from within the code part (the minted part) and you can refer in your main text to lines of code. On top of that, it provides a listings environment so that you can generate a "list of listings" (...


3

The Teuchos Unit test harness in Trilinos natively supports unit tests that use MPI. Things like controlling output from multiple processes and aggregating pass/fail over all processes is automatic. Take a look: http://trilinos.org/docs/dev/packages/teuchos/doc/html/group__Teuchos__UnitTest__grp.html


3

In Stodden, V. 2009. “The Legal Framework for Reproducible Scientific Research.” CiSE. Victoria Stodden recommends publishing the full "research compendium", and lists the following components on p. 38: The research paper The data - including documentation and code for processing the data The experiment — all source code; documentation, parameters, ...


3

For addressing the issue in the context of threads on a shared memory system, I've written this page that explains what we do in deal.II: http://dealii.org/developer/doxygen/deal.II/group__threads.html#MTWorkStream


2

I work for Elsevier. My company has started using the Collage framework (developed in response to the Executable Paper Grand Challenge) in journal issues to enable authors to publish chunks of executable code with their article. This feature makes it easier for readers to reproduce results reported in the article and to reuse published material for their own ...


2

I would like to point out that instead of using higher precision arithmetic for the addition, there's the possibility of using compensated summation (see [1]). This could increase the accuracy of the summation without the need to resort to larger data types. [1] Higham, N. J. The Accuracy of Floating Point Summation. SIAM Journal on Scientific Computing 14,...


2

I work for Elsevier. My company has started using the Collage framework (developed in response to the Executable Paper Grand Challenge) in journal issues to enable authors to include all of the data and code needed to reproduce the results and figures in their papers. This feature makes it easier for readers to reproduce results reported in the article and ...


Only top voted, non community-wiki answers of a minimum length are eligible