Local versus remote development

I know there are similar questions asked on other StackExchange sites, but I think development in Computational Science may work differently because of the computing environments that are common. I often can not run reasonable tests on my local machine due to time or memory constraints (or maybe I'm developing for intel coprocessors). I also can't really test distributed parallelism on my local machine, Thus I log in to a remote cluster and run my software there.

Developing locally is good because I usually have root on my own machine and can install nice tools like valgrind, gdb, libclang (for source code completion), ranger, and others. However, my computer likely has different compiler and library versions than are available on a remote machine (which is shared and I don't have privileges on).

Developing remotely ensures that compiler and library versions are the same and that I can edit, compile and run without having to sync my changes in some way. I can sync easily with git, but it can be a pain to commit, push and then pull just to fix a semicolon I forgot.

Ideally, I'd have all my tools on the remote machine, but without privileges some can be really annoying to install when they have several of dependencies and I can't use a package manager without root.

What's the best way to develop using a remote machine that I don't have privileges on?

• Theoretically, there are some package managers you can use without root privileges, such as Homebrew (OS X only), Linuxbrew (experimental), pkgsrc (multiple platforms), and HashDist (still in development). Using Homebrew on laptops has worked well; I can't speak to how this approach might work on a remote machine. – Geoff Oxberry Jun 17 '14 at 7:20
• Geoff, I'll look into some of these, they look interesting. – Keith Jun 17 '14 at 14:59

Ask your remote host to install what you need. We do this all the time for folks where I work. Typically they can help you out. Also, it's OK to do some of your development on your machine and then port your results to the remote machine.

• Thanks, Bill. Do those packages get installed system wide and become available to all users then (at least where you are, I think we are using similar setups where I am)? – Keith Jun 17 '14 at 14:59
• Typically. For us, it depends on what it is. If it's something obscure that you need help building, we might just walk you through an install in your directory, but if several groups ask for the same thing, we're likely to make a module available to all users through a central installation. – Bill Barth Jun 17 '14 at 16:16

I want to add something not mentioned yet here that might be useful (too long for a comment), because you say that syncing with git is a pain.

There is the tool sshfs (and I believe its osx equivalent is osxfuse). If your remote machine allows ssh connections, sshfs will be able to mount the remote filesystem locally over ssh. This is useful if, as I think you say, you can't mount remote directories already. Thus typically you would do this:

mkdir remote-sandbox
sshfs ssh://user@remote:~/sandbox remote-sandbox
cd remote-sandbox


and then edit the files in remote-sandbox directly. This would save you from having to synchronize files using git or rsync or scp, or what have you.

• Kirill, I've done this before, but in my experience it can be a bit laggy compared to true local development. I may try an rsync solution, but I'm not sure how that might interfere with git. – Keith Jun 19 '14 at 19:21