20

What you are asking for is the Elsivier grand challenge of the "Executable Paper". While many approaches have been tried, none are as compelling as the authors might suggest. Here are a few examples of techniques used. Madagascar Project takes your approach, inside the make script have the simulations run that produce the figures and paper simultaneously. ...


19

A lot has been written about how to design, execute, and report the results of computational experiments. This has obvious connections with open source software and the broader "open science" movement. Another important issue is the difference between "my code is faster than yours" research and research that helps us to better understand the properties of ...


17

In rough order of importance. Source Code Make the code that implements the key aspects of your algorithm available. Even if the user can't build or run it, they can read exactly what is done. I have several times noticed simple decisions that weren't documented in a paper, but which a couple minutes with the source code answered conclusively. Make it ...


15

A reduction implemented using MPI_Allreduce() is reproducible as long as you use the same number of processors, provided the implementation observed the following note appearing in Section 5.9.1 of the MPI-2.2 standard. Advice to implementors. It is strongly recommended that MPI_REDUCE be implemented so that the same result be obtained whenever the ...


14

As some comments have suggested, this approach has long been developed in the R community by building on Sweave and more recently, knitr. Obviously this approach has the disadvantage of being language-specific at the moment, but the advantage that its regularly used in academic papers. Use of Sweave in real publications The Journal of Biostatistics ...


14

To make more robust comparisons (on linux), you can : 1) On Intel CPUs the turbo overclocks your CPU. This is controlled by the temperature of the CPU, so it can behave differently from one run to the other. On Linux, you can block the frequency of the CPU as follows. For example, for 2.4GHz: echo 1 > /sys/module/processor/parameters/ignore_ppc for ...


13

I don't think that the definition of "reproducible research" requires that the author provide for free all of the tools that are needed to reproduce the results obtained. If some of it is proprietary, then it is incumbent on the future user, not the author, to make the arrangements to acquire the needed software. (You wouldn't expect to have to build the ...


13

I have not had a lot of success in using other people's solutions to this problem. I usually just want something simple that works for me and gets the job done. To this end, I generally try to write one python script which is in charge of running all the results, parsing the output, as well as building the figures/tables. I write my codes to generate ...


8

the planned longevity of TeX comes to mind: “Ever since those beginnings in 1977, the TeX research project that I embarked on was driven by two major goals. The first goal was quality: we wanted to produce documents that were not just nice, but actually the best. (…) The second major goal was archival: to create systems that would be independent of ...


8

I'm a happy user of GoogleTest with a C++ MPI code in a CMake/CTest build environment: CMake automatically installs/links googletest from svn! adding tests is a one-liner! writing the tests is easy! (and google mock is very powerful!) CTest can pass command-line parameters to your tests, and exports data to CDash! This is how it works. A batch of unit-...


7

Even more important, in my opinion, is making sure that you can figure out how to re-generate all your results from scratch in a month or a year (for instance, when referees ask you to add or modify something). For that purpose, what I do is include a text file with very detailed directions on how to reproduce all results. It's best if you test these out ...


7

You might consider writing the entire paper in Noweb. It's a bit tedious to set up, but it's a very powerful way to mix code and LaTeX-formatted text, equations, and figures. For long programs, it tends to turn your code into more of a book than an article, but for short programs, it might work out pretty well. If you don't want to go that far, it still ...


6

Emacs's orgmode in combination with Babel achieves that. Babel can execute code snippets from various programming and scripting languages, e.g., it could open the file containing the simulation data and put it into a table in orgmode, which can be exported to LaTeX (and many other formats). It takes quite a while getting used to all the key combos in orgmode,...


6

Most journals aren't set up for this in any formal way, but we've recently founded the Archive of Numerical Software that is specifically intended to have the source code and everything else that's necessary be part of the article. Check it out: http://journals.tdl.org/ans Submissions are welcome!


6

For the most part I ditto Jed's answer. However, there is a different way out: Given the size of normal floating point numbers, you can store every number in a 4000-or-so bit fixed point number. So if you do a reduction on floating point numbers thus embedded, you get an exact calculation, no matter the associativity. (Sorry, I don't have the reference to ...


6

In deal.II, we have a testsuite that is driven by a regular Unix Makefile. It has a default target that runs all the usual tests, and a separate target for expensive tests. Running each test is done using a generic rule but the default target calls the generic rule only for certain tests and the expensive target calls it for the expensive tests. Because it's ...


6

Any numerical differences between A and B will become exponentially larger with time (i.e. the Lyapunov instability, as discussed in Frenkel and Smit). Even a small difference due to basis set size could result in dramatic differences in the trajectory over time. So I'm not sure that a comparison between individual trajectories will be meaningful. It may be ...


5

You can implement a numerically stable reduction algorithm in MPI the same as you can do in serial. There may be a performance hit, of course. If you can afford to replicate the vector, just use MPI_Gather and do the numerically stable reduction in serial on the root. In some cases, you may find the performance hit is not a big deal. Another solution is ...


5

The noweb approach mentioned by Bill has evolved quite a bit, both in it's original spirit of documenting code (rather than scientific publication) under the term literate programming and now comes in many flavors (I guess noweb was a generalization of cweb initially), of which doxygen and various language specific versions can generate documentation in TeX, ...


5

I would've posted this in a comment, but I unfortunately don't have enough reputation. This answer (https://stackoverflow.com/a/42610074/9796552) should provide enough to help you achieve what you want. In short, you can use Julia's Sys module which contains functions dedicated to retrieving system information.


4

If you're using LaTeX, a relatively low-tech solution is to have your code spit out a file (or else use a script to filter from your code's output) containing a whole bunch of lines like this: \newcommand{\myresults1}{<value>} Then you can use the \input command to add that file into your document, and use the defined commands to place the values.


4

If running all your code is cheap then you could do something low-tech like the following: You could template your documents with formatted strings so that they look like this "we observed a %(fractional_improvement)s increase in ..." Have python scripts that look like this results = {"alpha" : run_alpha_computation(...), "...


4

There are many technical challenges that make exact bit-for-bit reproducibility of computational results extremely hard to achieve. At the software level, changes to the code or any of the libraries used by the code can obviously cause different results to be produced. You'd be surprised by the number of support libraries that can end up linked into a ...


4

I have never been of the opinion that "reproducible research" means that your fellow readers and scientists should only have to click "Run" (or compile or what have you) to entirely reproduce your results. If nothing else, that would kill the research of anyone who uses a proprietary statistical analysis package, or make things "unreproducable" if they were ...


4

There are several MPI-enabled software packages that use the CMake set of tools for testing. The ones that I can think of off the top of my head are Trilinos, VTK and ParaView. I would think that you don't want to assume that the executable needs to be launched with mpirun and/or mpiexec. CMake has support for specifying how to properly launch the executable ...


3

For addressing the issue in the context of threads on a shared memory system, I've written this page that explains what we do in deal.II: http://dealii.org/developer/doxygen/deal.II/group__threads.html#MTWorkStream


3

In Stodden, V. 2009. “The Legal Framework for Reproducible Scientific Research.” CiSE. Victoria Stodden recommends publishing the full "research compendium", and lists the following components on p. 38: The research paper The data - including documentation and code for processing the data The experiment — all source code; documentation, parameters, ...


3

The LaTeX package minted provides very extensive syntax highlighting (based on Pygments) and allows cross-referencing in both directions. You can escape to LaTeX from within the code part (the minted part) and you can refer in your main text to lines of code. On top of that, it provides a listings environment so that you can generate a "list of listings" (...


3

We simply roll our own code in deal.II -- in essence, we tell the framework to execute tests using mpirun -np .... We had previously just used a Makefile-based testing scheme (compile, link, execute test, then compare the output with one that had previously been saved) and you can find this here: https://svn.dealii.org/branches/releases/Branch-8-0/tests/mpi/...


3

The Teuchos Unit test harness in Trilinos natively supports unit tests that use MPI. Things like controlling output from multiple processes and aggregating pass/fail over all processes is automatic. Take a look: http://trilinos.org/docs/dev/packages/teuchos/doc/html/group__Teuchos__UnitTest__grp.html


Only top voted, non community-wiki answers of a minimum length are eligible