Cloud solutions for microscopy image analysis

Question

At our research facility we routinely perform what are called PALM (Photo-activated localisation microscopy) or STORM (stochastic optical reconstruction microscopy) experiments.

You can read all about it in the links above but in short it boils down to this:

1. Normal optical microscopy is diffraction limited. We can only resolve features bigger than say 200-300 nm because of the very nature of the wavelengths and optics we use.
2. We can circumvent this resolution limitation by finding ways to observe point emitters in a sample and using what is known about their "point spread function" to fit the location of those emitters with sub diffraction limit accuracy.
3. Observing many such point emitters in a sample allows us to reconstruct an image of the entire sample with enhanced resolution.

Practically, we achieve all this by recording long (50k frames) "movies" of samples using EM-CCD cameras which produce 16 bit 512*512 pixel images. As such, these datasets grow quite big, particularly if we run lots of experiments daily/weekly.

Next we run established fitting routines on these datasets, such as the ones implemented here.

Currently we do all this in what I call "Offline mode". We record these data sets, often dozens or hundreds of times a week. Store them on USB hard drives and run the above referenced analyses.

This leads to issues:

1. Yearly, we invest thousands of euros in disks because we need to store all erudite and might want to hang on to it for future reference. However, this data cannot be properly indexed and managed. Everyone uses their own schemes for naming/folders/...
2. Disks can go missing
3. ...

Reading so much about "cloud computing" lately, I wanted to try and find better solutions to the issue. I think that it should be possible to:

1. Install some kind of Dropbox like client on each measurement computer.
2. Upload all raw image stacks to a cloud storage facility
3. Expose that cloud storage as a virtual disk to nice software such as The Open microscopy Environment which already offers solutions for indexing/annotating ur type of data
4. Leverage the computational power of cloud systems to also speed up the analysis

In this respect I have contacted several cloud providers (MS Azure, Google, Amazon) but in my experience it is very hard to get in touch with someone knowledgable enough to first "get" our intended application and next to provide viable potential solutions.

Furthermore, although I grasp most of the concepts involved in modern cloud platforms I lack the time and in depth knowledge to roll my own solution.

Nonetheless I cannot imagine that we would be the first people to run into this issue.

Does anybody have an idea in this respect?

DISCLAIMER: It could be that this question is out of scope for this particular SA site but I did not seem to find another where this would be better suited (or I missed it).

Answer 1

Awesome to find another microscopist on this site. Welcome!

Short Answer:

There is no magic-bullet on the market that will solve this problem space for you.

Longer Answer:

Putting together a few commercailly available offerings gets you pretty far.

At my work we take ~1-10 thousand $4\,096\times25\,000$ images weekly, which puts us in a similar problem-scale space, so I wanted to share my experiences first in the form of some quick bullet points, then a more through system description of what we created to deal with this all.

• Compress Early. Investing time (lossless) compressing your images will save on network IO, storage costs, transfer times, and almost always will speed analysis. PALM and STORM are great candidates given typically low per-image entropy. We built a tiny cluster to get this all done. With 32 cores it still cost less than 2% of a new STORM setup :)
• Archive/Upload continuously. We use s3 for storage and have a constant background thread uploading imagery. Your ability to upload to AWS will be limited only by your local connection.
• Use s3 or similar competitive products. You really can not beat the storage costs of this service, unless you have some form of free/subsidized IT, electricity, or hard-drives. We use s3 buckets for each sample, and eventually migrate them to Glacier storage to reduce costs on already analyzed data.
• Leverage Cloud IO magic. Any particular computer pulling a single frame from AWS gets typically 5-15MB/s, however 500 computers pulling a frame each get the same rate, totaling in GB/s. This is really powerful for parallel analyses. Here is a great video about this with a microscopy focus.

Here is the system diagram that I drew for some documentation:

In short, we have a number of microscopes feeding a compression-cluster that all eventually converge to one IO endpoint that syncs with s3 into sample based buckets. This stack is not yet OS, but I can share it in a limited way fairly easily with anyone interested.

This is a specialized solution, but any fraction of this could be killed by a couple of shell-scripts with gnu parallel, image-magic, s3fs, and a reasonably sized compute cluster.

All this same stuff goes if you have a supporting HPC facility, it will just be cheaper :)

Answer 2

Based on our discussion above, I'd recommend the following: Find your local, regional, or national supercomputing, HPC, or research computing outfit and learn about their support for these kinds of data storage and computing activities. This kind of workload is one of the many things covered under the umbrella of "Big Data", and all of the supercomputing centers that I know of are moving to support these kinds of things. They are also being actively pushed to do so by their funding agencies.

In many cases, they will be able to support your workload for free or at a much reduced cost compared to commercial cloud options because they are subsidized to do so at the University, regional government, or national government level. Multi-terabyte (which this sounds like) data ingest or at-rest storage costs for commercial clouds are astronomical in comparison, and they typically don't have a mandate to support science.