Scholarly papers in scientific computing (and many other fields, nowadays) typically involve some amount of code or even whole software packages that were written specifically for that paper or were used to obtain results in the paper. What is the best way to help readers of the paper access the code? My current approach is to put a link to a Github repository (along with a particular version tag) in the paper or in a citation.
$\begingroup$
$\endgroup$
5
asked Jan 12, 2012 at 14:14
-
2$\begingroup$ Sharing the code is a great idea and should be done more. I know I could be better at providing the relevant code for a paper. A Github repo seems like a good solution. Certainly much better than including source code in an appendix, which I've seen done for smaller coding efforts. $\endgroup$– BarronCommented Jan 12, 2012 at 15:48
-
4$\begingroup$ This is a related MO question. $\endgroup$– J. M.Commented Jan 12, 2012 at 23:42
-
$\begingroup$ @J.M. Thanks, the answers on MO are very good! $\endgroup$– David KetchesonCommented Jan 13, 2012 at 19:27
-
$\begingroup$ note that you can publish ipython notebooks on github and they are rendered, except for interactive parts $\endgroup$– den.run.aiCommented Oct 30, 2015 at 17:15
-
1$\begingroup$ @denfromufa Unfortunately, Github disables Mathjax, so math is not rendered either. That makes it pretty useless for most relevant fields. But there is always nbviewer. $\endgroup$– David KetchesonCommented Oct 30, 2015 at 18:03
Add a comment
|
8 Answers
$\begingroup$
$\endgroup$
9
Well, I think you have a few options.
- If you have a stable page—such as one sponsored by a university or other non-profit institution that's unlikely to vanish anytime soon—you could publish there.
- You could use a service like Github or Bitbucket or SourceForge to distribute the code.
- If the code is of marginal general value (it's an analysis code for a specific set of conditions, etc.), you could make the code available as a "supplemental information" download with the paper in which you use it.
- You could use some combination of the above.
In any or all of these cases, however, you should indicate the sourcing clearly in the article, and indicate what kind of licensing it is (GPL, Creative Commons, etc.), so that there's no IP-related issues down the line.
-
7$\begingroup$ I think one should put one's code in the most likely place to survive, and in multiple places if possible. University pages seem less likely to survive than hosting services, for example. Having the journal make some snapshot available as well makes sense. Unfortunately, no journal that I know of does repos hosting. $\endgroup$ Commented Jan 13, 2012 at 6:50
-
1$\begingroup$ A student shouldn't probably put software on a personal home page; however, I would argue that for a typical research code, there's probably more to be gained by distributing it on a page associated with a research group than an external page where the attribution is likely to be lost. As for the journals, it's true that they don't do repository hosting. However, the ability to have "supplemental info" in the form of the research code I think satisfies most of the requirements of responsible scientific software development. (If need be.) $\endgroup$– aeismailCommented Jan 13, 2012 at 8:10
-
$\begingroup$ My feeling is that university pages are more likely to get lost than regular hosting sites. Of course, most of the hosting sites that are popular nowadays (Bitbucket, Github, Google Code) haven't been around that long. On the other hand, Sourceforge for example has been around for a while. $\endgroup$ Commented Jan 13, 2012 at 8:53
-
$\begingroup$ There are other issues to be aware of; IP concerns and university or government regulations may control the choice of repositories as well. But the counterargument is that there are a number of codes (NAMD being one major example) that have had successful distribution on university-owned sites. In general, the "significance" of the code will determine how visible it is. I doubt that a code that develops a significant user base will ever disappear completely. $\endgroup$– aeismailCommented Jan 13, 2012 at 9:35
-
1$\begingroup$ True, but if the code is obscure doesn't mean it is Ok if it disappears. And one would hope that most scientific code would be under a free license and without unreasonable restriction. I believe the NIH for example is now mandating this for work developed with NIH (taxpayer) money. I think this should be the case for all taxpayer-funded projects. $\endgroup$ Commented Jan 13, 2012 at 10:28
$\begingroup$
$\endgroup$
4
Great question and great replies, but I think none addresses the question of persistence adequately, if the goal is to achieve the same standard accorded to the publication itself. (Which may be silly given the chances the code still runs, but may still be at least as useful as the publication all the same).
University websites journal supplements aren't persistent
University websites are unlikely to provide the stability or the redundancy to preserve hosted content. Content is more difficult to cite and typically lacks machine readable metadata.
Unfortunately it appears that journals are not doing much better in maintaining their supplemental materials (see Anderson et al. 2006), and may not accept the necessary formats, or even accept supplemental material at all (see one notable example).
For these reasons, folks concerned with long term archiving of data have unanimously turned towards advocating for the use of dedicated repositories rather than websites or supplemental materials, and many journals now mandate this practice. It seems only fair that code be held to this standard.
The solution of many copies?
Github and related sites have yet to prove longevity over the 100s of years scale achieved by university libraries and established publishers. By facilitating widespread distribution it may provide a solution others have echoed in the comments, including one fellow who couldn't comment on stackexchange,
...let us save what remains: not by vaults and locks which fence them from the public eye and use in consigning them to the waste of time, but by such a multiplication of copies, as shall place them beyond the reach of accident.
— Thomas Jefferson, February 18, 1791
Figshare & the CLOCKSS standard
The only archival standard I am aware of is figshare, which can accept full code repositories (as "filesets" for the moment, but I believe will soon have the option to be listed as type "code"). The key piece to figshare is not only the citable DOI with programmatic metadata, but the backing of CLOCKSS archival service, which maintains copies of all its content at 12 geographically and geo-politically distributed nodes around the world. Should figshare go out of business or cease to exist, this will trigger all its content to be freely available from CLOCKSS.
Consequently, I'd suggest using Github for distribution of code, but also providing an archival copy to figshare at the time of publication.
-
1$\begingroup$ figshare is a great step forward, though the CC-BY license is not a software license, and I don't know how many scientists are willing to release their code under CC0, so this is an issue to address. I do appreciate that they use DOI and CLOCKSS, though, that is great. $\endgroup$ Commented Oct 10, 2012 at 17:50
-
$\begingroup$ Yeah, great point about the licenses still being somewhat problematic, particularly for more fully developed software. For scripts to replicate an analysis I could see CC0 being more appropriate. $\endgroup$– cboettigCommented Oct 10, 2012 at 18:09
-
$\begingroup$ Google code might be slightly better for the wider audience as you can have a nicer webpage with summary, images, DOI link, higher visibility in search etc. You should definitely put a tgz in the Download section and provide a link on the front page. Remember that majority of non-developers are not even familiar with version control let alone git/hg. Subversion is as far as I would go for a broader audience. $\endgroup$– staliCommented Oct 10, 2012 at 20:55
-
1$\begingroup$ @stali recall that github also supports custom webpages for repositories through gh-pages and downloadable tarballs from downloads. But neither Google nor Github provide a separate DOI for the code, nor address the archival longevity beyond the life of the company afaik. $\endgroup$– cboettigCommented Oct 10, 2012 at 22:37
$\begingroup$
$\endgroup$
1
You may use some fancy pdf-techniques to simply attach the code to the pdf (that is, the code-files are embedded into the pdf and can be "downloaded" with a click on some button in the pdf). This can be accomplished with the attachfile package, for example. Of course, this work with preprints (although I don't know if it already works with the arxiv) but you probably get problems with journal-files...
-
$\begingroup$ Very cool! I didn't know LaTeX could do this. $\endgroup$– qubyteCommented Jan 19, 2012 at 7:03
$\begingroup$
$\endgroup$
2
For small scripts that are specific to a specific research project, the best place for publication is the journal's Web site, as "supplementary information" to the paper. That's where it is easiest to find for someone who reads the article.
More substantial packages that are of interest for other projects as well should better be published separately. Unfortunately there is no really good solution at this time. Ideally, a code publication would be permanently accessible through a DOI, just like a paper, but I am not aware of any hosting site that hands out DOIs and guarantees their permanence. Public repositories like Github or Bitbucket are perhaps the best bet for now.
The best solution would be to publish the paper packaged with the code and the data that go with it, but that is not yet technically feasible. I am working on a research prototype exploring this idea, see this site for details.
-
1$\begingroup$ +1 for ActivePapers. I don't think it meets my needs now, but I'm glad to see someone working on a solution! $\endgroup$ Commented Jan 13, 2012 at 19:26
-
$\begingroup$ figshare provides DOIs: see figshare.com/blog/… $\endgroup$ Commented Jun 22, 2012 at 7:11
$\begingroup$
$\endgroup$
I've taken two tactics, born of the fact that I anticipate changing institutions soon, so my university URL isn't stable in the slightest.
When the code is relatively short, I've tried including it as a supplemental appendix in the journal itself, under the assumption that they'll probably do a decent job keeping the paper and the code in roughly the same place. This is especially useful for code where there isn't a wide amount of general interest - code that's somewhat useless without the paper in question to provide context.
But for source-code, actual software, and more complicated projects or ones of general interest, I've been following your tactic of linking to a GitHub repository, which should at least be stable for the average productive lifespan of my papers.
$\begingroup$
$\endgroup$
2
Take a look at http://www.runmycode.org. They host companion sites for code associated with research papers. If the code is R, Matlab, or a few others, it will actually run the code for you. I haven't tried it yet, but I intend to. I think David Donoho and his collaborators use it.
answered Jan 30, 2013 at 8:51
-
$\begingroup$ Ah. You've used this already. runmycode.org/CompanionSite/site.do?siteId=158 $\endgroup$ Commented Jan 30, 2013 at 8:56
-
$\begingroup$ @David Ketcheson and I also did an experiment in December using the wakari.io stack and IPython notebooks for one of our Python-based codes. You can check out the PyClaw reproducibility notebooks here. $\endgroup$ Commented Jan 30, 2013 at 12:34
$\begingroup$
$\endgroup$
University libraries could be a place for this or the hosting center of the university.
$\begingroup$
$\endgroup$
4
As a reader, a statement in the paper to the effect that code can be obtained by contacting the author directly would be effective. As an author, this could help foster collaboration, and give me an opportunity to remind people to cite my article if they use the code in their work.
-
4$\begingroup$ That's an interesting point of view, and I'm curious to see how common it is. Personally, it's what I'm trying to get away from. I feel that it's like publishing an incomplete paper and requiring readers to ask for the full thing. See sciencecodemanifesto.org . $\endgroup$ Commented Jan 13, 2012 at 14:34
-
2$\begingroup$ Having the contact address on one of my most prominent papers being essentially dead - and unsure about some others - I'm generally opposed to this as a solution. "Contacting me" isn't necessarily the easiest thing in the world to do - especially a decade later. $\endgroup$– FomiteCommented Jan 14, 2012 at 3:10
-
2$\begingroup$ The "contact me" approach also does not guarantee reproducibility. When you contact me asking for some code, I'd probably send you the most up-to-date version, not the one I used in the original paper. If only because I wouldn't have the original version any more. $\endgroup$– khinsenCommented Jan 15, 2012 at 10:15
-
3$\begingroup$ Empirical studies actually contacting an author and asking for data, even when the author has signed a license agreement to provide it on request, have shown that surprisingly few authors comply. For instance, see dx.doi.org/10.1371/journal.pone.0007078 and citations therein. If this doesn't work well for data, I suspect it is not a good solution for code either. $\endgroup$– cboettigCommented Jan 18, 2012 at 18:47