Take the 2-minute tour ×
Computational Science Stack Exchange is a question and answer site for scientists using computers to solve scientific problems. It's 100% free, no registration required.

At our research facility we routinely perform what are called PALM (Photo-activated localisation microscopy) or STORM (stochastic optical reconstruction microscopy) experiments.

You can read all about it in the links above but in short it boils down to this:

  1. Normal optical microscopy is diffraction limited. We can only resolve features bigger than say 200-300 nm because of the very nature of the wavelengths and optics we use.
  2. We can circumvent this resolution limitation by finding ways to observe point emitters in a sample and using what is known about their "point spread function" to fit the location of those emitters with sub diffraction limit accuracy.
  3. Observing many such point emitters in a sample allows us to reconstruct an image of the entire sample with enhanced resolution.

Practically, we achieve all this by recording long (50k frames) "movies" of samples using EM-CCD cameras which produce 16 bit 512*512 pixel images. As such, these datasets grow quite big, particularly if we run lots of experiments daily/weekly.

Next we run established fitting routines on these datasets, such as the ones implemented here.

Currently we do all this in what I call "Offline mode". We record these data sets, often dozens or hundreds of times a week. Store them on USB hard drives and run the above referenced analyses.

This leads to issues:

  1. Yearly, we invest thousands of euros in disks because we need to store all erudite and might want to hang on to it for future reference. However, this data cannot be properly indexed and managed. Everyone uses their own schemes for naming/folders/...
  2. Disks can go missing
  3. ...

Reading so much about "cloud computing" lately, I wanted to try and find better solutions to the issue. I think that it should be possible to:

  1. Install some kind of Dropbox like client on each measurement computer.
  2. Upload all raw image stacks to a cloud storage facility
  3. Expose that cloud storage as a virtual disk to nice software such as The Open microscopy Environment which already offers solutions for indexing/annotating ur type of data
  4. Leverage the computational power of cloud systems to also speed up the analysis

In this respect I have contacted several cloud providers (MS Azure, Google, Amazon) but in my experience it is very hard to get in touch with someone knowledgable enough to first "get" our intended application and next to provide viable potential solutions.

Furthermore, although I grasp most of the concepts involved in modern cloud platforms I lack the time and in depth knowledge to roll my own solution.

Nonetheless I cannot imagine that we would be the first people to run into this issue.

Does anybody have an idea in this respect?

DISCLAIMER: It could be that this question is out of scope for this particular SA site but I did not seem to find another where this would be better suited (or I missed it).

share|improve this question
    
Well, there's softwarerecs.stackexchange.com, although I don't see why a question about handling large datasets in scientific computing should be off-topic here. Of course, storing sensitive data (either in a business or -- for medical imaging -- privacy sense) on a system not under your control might be an issue. –  Christian Clason Jul 27 at 10:34
    
Hi @ChristianClason, thanks for the feedback. Indeed, data privacy concerns have also been raised by coworkers but I guess they could be addressed through some kind of encryption of the storage system. As I understand services like Dropbox also store their customers data on 3rd party providers (DB is on S3 I believe) –  Kris Jul 27 at 11:57
    
Yes, but it's not Dropbox's data they're storing, so they don't have to care... There's also SpiderOak, which promises client-side encryption, and OwnDrive, which is an open source framework you can install on your own servers (or possibly on top of S3 etc.). –  Christian Clason Jul 27 at 12:15
    
What country? Do you have access to a supercomputing facility at your university, state/region, or national-scale facility that could help you? Most of these have moved into supporting problems like this (often at no charge) and not just traditional, simulation-based computational science. –  Bill Barth Jul 27 at 12:34
    
I don't know exactly about Belgium, thought I have had some interactions. At my University, the supercomputing/HPC/bigdata people are separate from IT. Have a look around your University for someone that does one of those things or maybe "research computing". Maybe talk to these guys. –  Bill Barth Jul 27 at 12:43

2 Answers 2

Awesome to find another microscopist on this site. Welcome!

Short Answer:

There is no magic-bullet on the market that will solve this problem space for you.

Longer Answer:

Putting together a few commercailly available offerings gets you pretty far.

At my work we take ~1-10 thousand $4\,096\times25\,000$ images weekly, which puts us in a similar problem-scale space, so I wanted to share my experiences first in the form of some quick bullet points, then a more through system description of what we created to deal with this all.

  • Compress Early. Investing time (lossless) compressing your images will save on network IO, storage costs, transfer times, and almost always will speed analysis. PALM and STORM are great candidates given typically low per-image entropy. We built a tiny cluster to get this all done. With 32 cores it still cost less than 2% of a new STORM setup :)
  • Archive/Upload continuously. We use s3 for storage and have a constant background thread uploading imagery. Your ability to upload to AWS will be limited only by your local connection.
  • Use s3 or similar competitive products. You really can not beat the storage costs of this service, unless you have some form of free/subsidized IT, electricity, or hard-drives. We use s3 buckets for each sample, and eventually migrate them to Glacier storage to reduce costs on already analyzed data.
  • Leverage Cloud IO magic. Any particular computer pulling a single frame from AWS gets typically 5-15MB/s, however 500 computers pulling a frame each get the same rate, totaling in GB/s. This is really powerful for parallel analyses. Here is a great video about this with a microscopy focus.

Here is the system diagram that I drew for some documentation: A look at our internals

In short, we have a number of microscopes feeding a compression-cluster that all eventually converge to one IO endpoint that syncs with s3 into sample based buckets. This stack is not yet OS, but I can share it in a limited way fairly easily with anyone interested.

This is a specialized solution, but any fraction of this could be killed by a couple of shell-scripts with gnu parallel, image-magic, s3fs, and a reasonably sized compute cluster.

All this same stuff goes if you have a supporting HPC facility, it will just be cheaper :)

share|improve this answer
    
This most definitely sounds interesting! I'll take some time to process this but I'll definitely get back to you. Thanks already! –  Kris Jul 28 at 7:01

Based on our discussion above, I'd recommend the following: Find your local, regional, or national supercomputing, HPC, or research computing outfit and learn about their support for these kinds of data storage and computing activities. This kind of workload is one of the many things covered under the umbrella of "Big Data", and all of the supercomputing centers that I know of are moving to support these kinds of things. They are also being actively pushed to do so by their funding agencies.

In many cases, they will be able to support your workload for free or at a much reduced cost compared to commercial cloud options because they are subsidized to do so at the University, regional government, or national government level. Multi-terabyte (which this sounds like) data ingest or at-rest storage costs for commercial clouds are astronomical in comparison, and they typically don't have a mandate to support science.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.