Tell me more ×
Computational Science Stack Exchange is a question and answer site for scientists using computers to solve scientific problems. It's 100% free, no registration required.
up vote 32 down vote favorite
9

Scholarly papers in scientific computing (and many other fields, nowadays) typically involve some amount of code or even whole software packages that were written specifically for that paper or were used to obtain results in the paper. What is the best way to help readers of the paper access the code? My current approach is to put a link to a Github repository (along with a particular version tag) in the paper or in a citation.

share|improve this question
2  
Sharing the code is a great idea and should be done more. I know I could be better at providing the relevant code for a paper. A Github repo seems like a good solution. Certainly much better than including source code in an appendix, which I've seen done for smaller coding efforts. – Barron Jan 12 '12 at 15:48
4  
This is a related MO question. – J. M. Jan 12 '12 at 23:42
@J.M. Thanks, the answers on MO are very good! – David Ketcheson Jan 13 '12 at 19:27

8 Answers

up vote 15 down vote accepted

Well, I think you have a few options.

  1. If you have a stable page—such as one sponsored by a university or other non-profit institution that's unlikely to vanish anytime soon—you could publish there.
  2. You could use a service like Github or Bitbucket or SourceForge to distribute the code.
  3. If the code is of marginal general value (it's an analysis code for a specific set of conditions, etc.), you could make the code available as a "supplemental information" download with the paper in which you use it.
  4. You could use some combination of the above.

In any or all of these cases, however, you should indicate the sourcing clearly in the article, and indicate what kind of licensing it is (GPL, Creative Commons, etc.), so that there's no IP-related issues down the line.

share|improve this answer
6  
I think one should put one's code in the most likely place to survive, and in multiple places if possible. University pages seem less likely to survive than hosting services, for example. Having the journal make some snapshot available as well makes sense. Unfortunately, no journal that I know of does repos hosting. – Faheem Mitha Jan 13 '12 at 6:50
1  
A student shouldn't probably put software on a personal home page; however, I would argue that for a typical research code, there's probably more to be gained by distributing it on a page associated with a research group than an external page where the attribution is likely to be lost. As for the journals, it's true that they don't do repository hosting. However, the ability to have "supplemental info" in the form of the research code I think satisfies most of the requirements of responsible scientific software development. (If need be.) – aeismail Jan 13 '12 at 8:10
My feeling is that university pages are more likely to get lost than regular hosting sites. Of course, most of the hosting sites that are popular nowadays (Bitbucket, Github, Google Code) haven't been around that long. On the other hand, Sourceforge for example has been around for a while. – Faheem Mitha Jan 13 '12 at 8:53
There are other issues to be aware of; IP concerns and university or government regulations may control the choice of repositories as well. But the counterargument is that there are a number of codes (NAMD being one major example) that have had successful distribution on university-owned sites. In general, the "significance" of the code will determine how visible it is. I doubt that a code that develops a significant user base will ever disappear completely. – aeismail Jan 13 '12 at 9:35
1  
True, but if the code is obscure doesn't mean it is Ok if it disappears. And one would hope that most scientific code would be under a free license and without unreasonable restriction. I believe the NIH for example is now mandating this for work developed with NIH (taxpayer) money. I think this should be the case for all taxpayer-funded projects. – Faheem Mitha Jan 13 '12 at 10:28
show 2 more comments
up vote 8 down vote

Great question and great replies, but I think none addresses the question of persistence adequately, if the goal is to achieve the same standard accorded to the publication itself. (Which may be silly given the chances the code still runs, but may still be at least as useful as the publication all the same).

University websites journal supplements aren't persistent

University websites are unlikely to provide the stability or the redundancy to preserve hosted content. Content is more difficult to cite and typically lacks machine readable metadata.

Unfortunately it appears that journals are not doing much better in maintaining their supplemental materials (see Anderson et al. 2006), and may not accept the necessary formats, or even accept supplemental material at all (see one notable example).

For these reasons, folks concerned with long term archiving of data have unanimously turned towards advocating for the use of dedicated repositories rather than websites or supplemental materials, and many journals now mandate this practice. It seems only fair that code be held to this standard.

The solution of many copies?

Github and related sites have yet to prove longevity over the 100s of years scale achieved by university libraries and established publishers. By facilitating widespread distribution it may provide a solution others have echoed in the comments, including one fellow who couldn't comment on stackexchange,

...let us save what remains: not by vaults and locks which fence them from the public eye and use in consigning them to the waste of time, but by such a multiplication of copies, as shall place them beyond the reach of accident.

— Thomas Jefferson, February 18, 1791

Figshare & the CLOCKSS standard

The only archival standard I am aware of is figshare, which can accept full code repositories (as "filesets" for the moment, but I believe will soon have the option to be listed as type "code"). The key piece to figshare is not only the citable DOI with programmatic metadata, but the backing of CLOCKSS archival service, which maintains copies of all its content at 12 geographically and geo-politically distributed nodes around the world. Should figshare go out of business or cease to exist, this will trigger all its content to be freely available from CLOCKSS.

Consequently, I'd suggest using Github for distribution of code, but also providing an archival copy to figshare at the time of publication.

share|improve this answer
1  
figshare is a great step forward, though the CC-BY license is not a software license, and I don't know how many scientists are willing to release their code under CC0, so this is an issue to address. I do appreciate that they use DOI and CLOCKSS, though, that is great. – Aron Ahmadia Oct 10 '12 at 17:50
Yeah, great point about the licenses still being somewhat problematic, particularly for more fully developed software. For scripts to replicate an analysis I could see CC0 being more appropriate. – cboettig Oct 10 '12 at 18:09
Google code might be slightly better for the wider audience as you can have a nicer webpage with summary, images, DOI link, higher visibility in search etc. You should definitely put a tgz in the Download section and provide a link on the front page. Remember that majority of non-developers are not even familiar with version control let alone git/hg. Subversion is as far as I would go for a broader audience. – stali Oct 10 '12 at 20:55
1  
@stali recall that github also supports custom webpages for repositories through gh-pages and downloadable tarballs from downloads. But neither Google nor Github provide a separate DOI for the code, nor address the archival longevity beyond the life of the company afaik. – cboettig Oct 10 '12 at 22:37
up vote 4 down vote

You may use some fancy pdf-techniques to simply attach the code to the pdf (that is, the code-files are embedded into the pdf and can be "downloaded" with a click on some button in the pdf). This can be accomplished with the attachfile package, for example. Of course, this work with preprints (although I don't know if it already works with the arxiv) but you probably get problems with journal-files...

share|improve this answer
Very cool! I didn't know LaTeX could do this. – Mark S. Everitt Jan 19 '12 at 7:03
up vote 4 down vote

For small scripts that are specific to a specific research project, the best place for publication is the journal's Web site, as "supplementary information" to the paper. That's where it is easiest to find for someone who reads the article.

More substantial packages that are of interest for other projects as well should better be published separately. Unfortunately there is no really good solution at this time. Ideally, a code publication would be permanently accessible through a DOI, just like a paper, but I am not aware of any hosting site that hands out DOIs and guarantees their permanence. Public repositories like Github or Bitbucket are perhaps the best bet for now.

The best solution would be to publish the paper packaged with the code and the data that go with it, but that is not yet technically feasible. I am working on a research prototype exploring this idea, see this site for details.

share|improve this answer
1  
+1 for ActivePapers. I don't think it meets my needs now, but I'm glad to see someone working on a solution! – David Ketcheson Jan 13 '12 at 19:26
figshare provides DOIs: see figshare.com/blog/… – Jeromy Anglim Jun 22 '12 at 7:11
up vote 3 down vote

I've taken two tactics, born of the fact that I anticipate changing institutions soon, so my university URL isn't stable in the slightest.

When the code is relatively short, I've tried including it as a supplemental appendix in the journal itself, under the assumption that they'll probably do a decent job keeping the paper and the code in roughly the same place. This is especially useful for code where there isn't a wide amount of general interest - code that's somewhat useless without the paper in question to provide context.

But for source-code, actual software, and more complicated projects or ones of general interest, I've been following your tactic of linking to a GitHub repository, which should at least be stable for the average productive lifespan of my papers.

share|improve this answer
up vote 2 down vote

Take a look at http://www.runmycode.org. They host companion sites for code associated with research papers. If the code is R, Matlab, or a few others, it will actually run the code for you. I haven't tried it yet, but I intend to. I think David Donoho and his collaborators use it.

share|improve this answer
Ah. You've used this already. runmycode.org/CompanionSite/site.do?siteId=158 – Paul G. Constantine Jan 30 at 8:56
@David Ketcheson and I also did an experiment in December using the wakari.io stack and IPython notebooks for one of our Python-based codes. You can check out the PyClaw reproducibility notebooks here. – Aron Ahmadia Jan 30 at 12:34
up vote 0 down vote

University libraries could be a place for this or the hosting center of the university.

share|improve this answer
up vote -2 down vote

As a reader, a statement in the paper to the effect that code can be obtained by contacting the author directly would be effective. As an author, this could help foster collaboration, and give me an opportunity to remind people to cite my article if they use the code in their work.

share|improve this answer
4  
That's an interesting point of view, and I'm curious to see how common it is. Personally, it's what I'm trying to get away from. I feel that it's like publishing an incomplete paper and requiring readers to ask for the full thing. See sciencecodemanifesto.org . – David Ketcheson Jan 13 '12 at 14:34
2  
Having the contact address on one of my most prominent papers being essentially dead - and unsure about some others - I'm generally opposed to this as a solution. "Contacting me" isn't necessarily the easiest thing in the world to do - especially a decade later. – EpiGrad Jan 14 '12 at 3:10
2  
The "contact me" approach also does not guarantee reproducibility. When you contact me asking for some code, I'd probably send you the most up-to-date version, not the one I used in the original paper. If only because I wouldn't have the original version any more. – khinsen Jan 15 '12 at 10:15
3  
Empirical studies actually contacting an author and asking for data, even when the author has signed a license agreement to provide it on request, have shown that surprisingly few authors comply. For instance, see dx.doi.org/10.1371/journal.pone.0007078 and citations therein. If this doesn't work well for data, I suspect it is not a good solution for code either. – cboettig Jan 18 '12 at 18:47

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.