11
$\begingroup$

Is there a published list of best practices to ensure longevity of code, with an eye towards reproducible scientific results? (e.g. open source, documentation practices, selecting dependencies, selecting a language, virtual machines, etc).

Know of any studies (or lacking that, examples/anecdotes) that have tried to estimate the half-life of typical scientific code or other software (if that's even a reasonable question?)

Improve this question
$\endgroup$
1
8
$\begingroup$

the planned longevity of TeX comes to mind:

“Ever since those beginnings in 1977, the TeX research project that I embarked on was driven by two major goals. The first goal was quality: we wanted to produce documents that were not just nice, but actually the best. (…) The second major goal was archival: to create systems that would be independent of changes in printing technology as much as possible. When the next generation of printing devices came along, I wanted to be able to retain the same quality already achieved, instead of having to solve all the problems anew. I wanted to design something that would be still usable in 100 years. ” – Donald E. Knuth: Digital Typography, p. 559 (quoted from http://de.wikipedia.org/wiki/TeX )

Based on Knuth's books about digital typography, even a complete reimplementation of TeX and METAFONT should be possible. They include annotations and explanations for all the code.

By demanding that your results should be stable over decades you get into a kind of freezing dilemma. On one hand, you want to make it easy reproduce your results 100%, so you freeze your software / environment. On the other hand, someone who is interested in reproducing your results in the future will certainly want to build on it. This person will be stuck with very old software, making it very hard to change anything. For anything that builds on several external packages, already a few years are enough to make things practically unchangeable.

For TeX, freezing is announced in the 1990 article

The future of TEX and METAFONT http://www.ntg.nl/maps/05/34.pdf

"I strongly believe that an unchanging system has great value, even though it is axiomatic that any complex system can be improved. Therefore I believe that it is unwise to make further “improvements” to the systems called TEX and METAFONT. Let us regard these systems as fixed points, which should give the same results 100 years from now that they produce today."

The ideal system would combine reproducibility with changeability. Trying to be as self-contained, simple and well-tested as possible certainly helps.

Pardon me if I was disgressing too much from the original question. [cross posted from 'Scientists for Reproducible Research', reproducible-research@googlegroups.com ]

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for bringing this over Matthias. And welcome to scicomp! $\endgroup$
    – Aron Ahmadia
    Oct 10 '12 at 20:01
  • 2
    $\begingroup$ I think that the TeX example is actually not a very good one even though it's generally considered the classical case for a frozen system. The reason I think so is that nobody uses TeX directly any more. People use latex together with its infinity of packages and they are very much not frozen. As a consequence, I think that (La)TeX documents are as much subject to change as everything else. To me, TeX is like a virtual machine -- you may keep that frozen but as long as the code built on top of it keeps changing, nothing is won. $\endgroup$
    – Wolfgang Bangerth
    Oct 11 '12 at 1:55
  • $\begingroup$ Thanks, I think this is an excellent case-study from the point of view of software development, which may be rather different than the scientific point of view. The fact that everyone needs to build on TeX indirectly may be non-ideal for widely used software, but may be ideal demonstration that scientific code could still run successfully and be built upon decades later. But surely Knuth did more simply avoiding changes & updates in order to pursue 100 year stability? $\endgroup$
    – cboettig
    Oct 11 '12 at 19:59
4
$\begingroup$

There are many technical challenges that make exact bit-for-bit reproducibility of computational results extremely hard to achieve.

At the software level, changes to the code or any of the libraries used by the code can obviously cause different results to be produced. You'd be surprised by the number of support libraries that can end up linked into a typical scientific code.

At a lower level, recompiling any of the code or any of the libraries used by the code with a new compiler or with different compiler optimizations turned on can also cause problems. One reason is that various operations in the code might be performed in a different order when the code is recompiled. Since floating point addition is not associative (a+b)+c <> a+(b+c), this can give different results.

OK, so what if we preserve the entire software environment (OS, libraries, and compiled code) by (e.g.) burning it on to a bootable CD-Rom that will run the code. Now can we be sure that we'll get the same results if we run this code on a different computer?

Surprisingly, some codes actually vary the order of computations based on aspects of the particular processor model that they're running on. For example, optimized linear algebra libraries typically break up matrix multiplications to work on blocks that will fit into cache. When Intel releases a new microprocessor with a bigger cache the code might dynamically adjust the block size, resulting in arithmetic that is performed in a different order and giving different results. Other codes dynamically adjust the order of computations based on the amount of available memory- if you run the code on a computer with more memory that could well cause the arithmetic to be done in a different order and thus give different results.

Things get amazingly more complicated when you throw in multithreaded code, since the exact execution history of the different threads is often non-deterministic and this can again cause arithmetic operations to be performed in a different order from one run to the next.

In practice the most that you can really hope for are results that are similar from one machine to the next, up to the accuracy tolerances of the algorithms used. e.g. if I have a root finding problem and use bisection to get a root to within +-1.0e-10, then I should be happy as long as different machines are producing answers that agree within that tolerance.

Improve this answer
$\endgroup$
1
  • $\begingroup$ By the way, the issue with different compiler versions explains why it really isn't sufficient to distribute a "frozen" version of the source code- the compiled code that is produced can vary depending on what version of the compiler is used and this can lead to different results. $\endgroup$
    – Brian Borchers
    Oct 11 '12 at 3:22
2
$\begingroup$

There have been many attempts at making reproducibility happen and there is a whole literature on this topic. My personal opinion from 15 years of scientific software is that it's unrealistic, as unsatisfactory as I find that answer. The problems are that (i) complex software has bugs and so can't be frozen; (ii) software is never feature complete and so development continues; (iii) what is the value of delivering with a paper several hundred thousands of lines of code?

As I say, I find this answer unsatisfactory. I believe that as a field, computational science has not been very successful in producing literature that instills trust that the results we publish are correct and reproducible. At the same time, I can't really come up with ways to do things better. For sure, releasing source code that goes with a paper is useful. At the same time, everyone who is honest will agree that the results in a paper will typically be produced by different versions of the code that, in most cases contain hacks describing different boundary conditions, different right hand sides, etc. A paper would then come with different versions of the same code. This is awkward for the reader to begin with, but it is outright unproductive if the code is large as it often happens today -- my two most recent papers used codes that are about 20,000 lines of code and that build on deal.II (600,000 lines of code) and Trilinos (1.5M lines of code). What information does that provide to a potential reader? (I should say that my codes are nevertheless available.)

Improve this answer
$\endgroup$
5
  • 2
    $\begingroup$ I'm less pessimistic but still unsatisfied. You could easily report the revision control tag or revision number associated with the code that produced the results in any given paper, and a totally scrupulous author would rerun all the results important to a given article with one code base. I don't think you need to deliver the code itself if a revision control system is in place, is publicly accessible, and the tags are published. $\endgroup$
    – Bill Barth
    Oct 11 '12 at 2:25
  • $\begingroup$ Sure, you could do that. The question is simply what a reader would do with the mass of code you throw at her. Yes, you can run it and verify that the results are the same as the ones shown. But what does that demonstrate? How is anyone going to verify -- in actual practice, not in theory -- that the results are correct? $\endgroup$
    – Wolfgang Bangerth
    Oct 11 '12 at 11:42
  • $\begingroup$ No, that's the part I'm in complete agreement with. Unless I think that you're an unscrupulous person, I don't need to rerun your code to reproduce the answers exactly. I think the bigger question is whether you've sufficiently demonstrated that you've verified your implementation and whether or not that can be validated against experiments. $\endgroup$
    – Bill Barth
    Oct 11 '12 at 13:57
  • $\begingroup$ Thanks, but I feel this does not address the question. There is certainly ample room to debate why having code available 15 years later is useful, but in this question I am simply asking if that code would still run for most people, given that you did archive it. I'm familiar with the literature encouraging code archiving, but no one encouraged a global archive for punch cards 40 years ago. Has technology increased or decreased the half-life of software? If archived code goes the way of the telegraph in 5 years time, the other issues are mute anyway. $\endgroup$
    – cboettig
    Oct 11 '12 at 19:50
  • $\begingroup$ I'm fairly sure you can get code written 15 years ago to run today, if with a good amount of work. I'm confident that you can get well-written codes from today to run in 15 years. $\endgroup$
    – Wolfgang Bangerth
    Oct 12 '12 at 1:24
2
$\begingroup$

For a possible solution to this problem, see my ActivePapers project. In summary, it describes how data and code can be packaged together with explicit dependencies on specific versions of each software component. This makes it possible to exactly reproduce a computation, while also permitting to run updated software on the same data.

I should add that ActivePapers is no more than a proof of concept and unlikely to be of any practical use in the near future. The reason is that it is based on the principle that all executable code must exist as JVM bytecode. At the moment, this excludes too many popular scientific libraries. However, once reproducibility is recognized as important, priorities in programming tools may change as well.

Improve this answer
$\endgroup$
1
$\begingroup$

I believe that as far as the choice of language goes, using a standardized one (e.g. C/Fortran/C++) would qualify as "best practice". If a package depends on 10 other libs/packages, especially those written in obscure languages then that obviously is bad for longevity. Many projects end up being orphaned after some time. I dont think major libs/api's like BLAS/LAPACK, PETSc, FFTW, MPI etc. would disappear anytime soon. BLAS is already pretty old.

The following piece of code (stolen from http://www.math.utah.edu/software/c-with-fortran.html) predates Fortran 77, uses Hollerith constants for char manipulation but compiles just fine 40-50 years later with the GNU Fortran Compiler:

stali@x61:~$ cat olde.f

       CALL S(12HHello, world, 12)
       END
       SUBROUTINE S(MSG,N)
       INTEGER K, N, M
       INTEGER MSG(1)
       M = (N + 3) / 4
       WRITE (6,'(20A4)') (MSG(K), K = 1,M)
       END

stali@x61:~$ gfortran -std=legacy olde.f; ./a.out
Hello, world

Open sourcing/putting it up somewhere like googlecode which is less likely to disappear soon (though they did shut down code search) is a no brainer.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the example! I'd be curious to see comparisons in other languages, including scripting languages -- do the first codes ever written in perl, python, or R still run with the same results? Are they more likely to do so or less likely than C or Fortran? $\endgroup$
    – cboettig
    Oct 11 '12 at 20:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.