7
$\begingroup$

We would like to reach Molecular Dynamics simulation of proteins with around 20000 atoms in explicit water with trajectories of around 1 microsecond each. We are looking at different options for computer resources to complete these simulations.

Since we (in Europe) can not apply for supercomputing time in Anton (from D.E.Shaw Research) and we have some funds (up to 500k €) we wonder which would be the best cluster or HPC infrastructure to buy for such calculations.

$\endgroup$
6
  • $\begingroup$ What scientific question are you trying to answer? I do not believe that microsecond MD answers many scientific questions. $\endgroup$ Oct 5, 2012 at 1:31
  • 2
    $\begingroup$ If you study protein folding, you should be aware that many proteins fold in the micro and even milisecond timescales, and some others even more. And for ligand-protein induced conformational changes, it falls into similar time domains. Do you need specific bibliographic references? $\endgroup$ Oct 5, 2012 at 3:13
  • $\begingroup$ I posted an answer, but before continuing, is the simulation 20K atoms before water is included? If so, how much water will you need to use? $\endgroup$
    – aeismail
    Oct 5, 2012 at 4:47
  • $\begingroup$ let's say protein + water around 100000 atoms $\endgroup$ Oct 5, 2012 at 7:15
  • $\begingroup$ You may want to decide first what software you want to use before making a decision on hardware. Although you can't request time from Anton, what's keeping you from requesting time at JuGene, for instance? $\endgroup$ Oct 5, 2012 at 18:24

4 Answers 4

12
$\begingroup$

For such a small simulation, I would strongly suggest looking into GPU-based solutions. This is probably what will get you the most ns/day/Euro.

In my opinion, the fastest fully-featured GPU-based Molecular Dynamics (MD) software out there is ACEMD (see here for timings). The software, however, is commercial, but has a single-GPU free version that can be used for evaluation purposes.

Other fully-featured, yet open-source, GPU-enabled MD packages include NAMD and GROMACS 4.6. Other projects include FenZi, but they don't seem to make their code available.

On the Joint Amber-Charmm (JAC) Benchmark, which consists of 23 558 atoms, but with a relatively short cutoff of 0.9nm, all these codes will get a handful of ns per day on a commodity GPU. That's still a few days of computing to get 1ms, but not bad considering that it's just one single machine.

$\endgroup$
12
  • $\begingroup$ @aeismail: I appreciate your edit, but LAMMPS is probably not the right tool for protein simulations, and HOOMD does not seem to do complex systems such as biomolecules at all. I have, however, added FenZi as another full-featured simulation package for proteins. $\endgroup$
    – Pedro
    Oct 5, 2012 at 10:15
  • $\begingroup$ Why do you think LAMMPS is inadequate for proteins, @Pedro? $\endgroup$ Oct 5, 2012 at 18:22
  • $\begingroup$ The usual arguments are that NAMD and Gromacs are faster than LAMMPS, and have some features (including allowing SHAKE along a backbone) that LAMMPS does not. That said, for the specific case of GPU-based processing, LAMMPS and HOOMD are superior options. $\endgroup$
    – aeismail
    Oct 5, 2012 at 19:37
  • $\begingroup$ @Deathbreath: LAMMPS was initially not designed for protein simulation and while it can do proteins today, it doesn't have a very large following in that field, and thus lacks the ecosystem of tools found in both GROMACS and NAMD (see this discussion for an example). I have also yet to be convinced that LAMMPS is any faster. $\endgroup$
    – Pedro
    Oct 8, 2012 at 9:35
  • $\begingroup$ @aeismail: I suppose you have references to back-up that claim? I have yet to see a direct comparison of HOOMD against anything but LAMMPS, less so for the specific case of simulating biomolecules. $\endgroup$
    – Pedro
    Oct 8, 2012 at 9:37
4
$\begingroup$

Even though you have so much money available to spend on computing resources, the bigger issue, as Pedro points out, is that your problem is relatively small. With roughly 100,000 atoms, your "sweet spot" on CPU's will likely be about 100 cores. If you try to use more than that, you'll end up spending a lot of time communicating information between processors, which can be much more expensive. You could try to purchase multithreaded processors, but they might not help nearly as much.

However, you are relatively speaking in the sweet spot for GPU computing. So your best bet would be either to use a package like HOOMD, or to take advantage of shared-memory machines using the multithreaded package options in codes such as LAMMPS or NAMD to avoid some of the internal message passing.

$\endgroup$
2
$\begingroup$

For all-atom, explicit solvent, bio-molecular systems of O(100k) atoms you ought now to be using GPU-accelerated codes. Even without knowing exactly the setup of your simulations it is most probable that ACEMD, AMBER, Gromacs, NAMD would all be adequate for your needs.

Generally these codes won't scale beyond a single system for your simulation size (unless with network like Infiniband), or even a few GPUs, and strongly favour GPU performance over CPU, so focus on machine configurations with several high-performance GPUs and good PCIe connectivity. Plan for 1-2 core/GPU. With some codes, there's no need to have multi-CPU systems, since the computation is done on the GPU (note that GROMACS will use both CPU and GPU effectively, so their quality should be balanced), nor employ a high performance interconnect, such as Infiniband.

All the codes use CUDA so Nvidia GPUs are the way to go. Geforce cards are perfectly adequate (eg the 4GB Geforce GTX680), and substantially more economical than the Teslas.

We sell a workstation optimised for ACEMD and other MD codes Acellera Metrocubo. Alternatively, register for the NVidia GPU Test Drive to be put in touch with other suitable hardware resellers.

With regard to the criticism of hydrogen mass re-partitioning, the theoretical and technical basis was first described in:

Improving efficiency of large time-scale molecular dynamics simulations of hydrogen-rich systems Feenstra et al. JCC 1999

doi://10.1002/(SICI)1096-987X(199906)20:8<786::AID-JCC5>3.0.CO;2-B

It is a widely used method, implemented not only in ACEMD but also Gromacs and, recently, Anton:

Atomic-level description of ubiquitin folding, Piana et al, PNAS (2013)

doi://10.1073/pnas.1218321110

$\endgroup$
1
  • $\begingroup$ how did you now people were talking about ACEMD? $\endgroup$ Apr 15, 2013 at 15:51
1
$\begingroup$

A deterministic code is not necessarily reversible, so it should not actually add any benefit in terms of statistical sampling, it kinds of produce the same errors but consistently. The consistency is useful for debugging. All codes are numerically integrated, so they will all drift sooner or later from the constant energy. It is important to stay close enough to the energy surface even when sampling in NVT and all codes used by people actually do it using mixed or fixed precision (NAMD, LAMMPS, ACEMD, AMBER, GROMACS, DESMOND, ANTON).

A single gtx680 would produce around 160 ns/day. A single gtx Titan 220 ns/day. So you get 1 microsecond in 4 days with the free version of acemd.

$\endgroup$
3
  • $\begingroup$ gianni, Welcome to SciComp! It appears that this answer is more of a comment on discussion on another answer rather than an answer of its own accord, and it doesn't address the original question which asked about hardware rather than software. It would be helpful if you edited the answer to include hardware recommendations for your suggested codes that would be efficient for this size of problem. $\endgroup$ Apr 15, 2013 at 12:10
  • $\begingroup$ Done. just edited the note above. $\endgroup$
    – gianni
    Apr 15, 2013 at 14:32
  • $\begingroup$ @gianni: Welcome to SciComp! Your answer is still mostly a software recommendation and a comment on another answer, and doesn't give a substantive hardware recommendation for a cluster (you mention a single sentence about performance on a single card, with no reference to system size, so I'm skeptical). Please answer the question more substantially, ask to convert your answer to a comment (in which case I'll remove the part about graphics cards), or delete your answer. $\endgroup$ Apr 16, 2013 at 3:28

Not the answer you're looking for? Browse other questions tagged or ask your own question.