Computational Science Stack Exchange is a question and answer site for scientists using computers to solve scientific problems. It only takes a minute to sign up.
Sign up to join this community
We would like to reach Molecular Dynamics simulation of proteins with around 20000 atoms in explicit water with trajectories of around 1 microsecond each. We are looking at different options for computer resources to complete these simulations.
Since we (in Europe) can not apply for supercomputing time in Anton (from D.E.Shaw Research) and we have some funds (up to 500k €) we wonder which would be the best cluster or HPC infrastructure to buy for such calculations.
For such a small simulation, I would strongly suggest looking into GPU-based solutions. This is probably what will get you the most ns/day/Euro.
In my opinion, the fastest fully-featured GPU-based Molecular Dynamics (MD) software out there is ACEMD (see here for timings). The software, however, is commercial, but has a single-GPU free version that can be used for evaluation purposes.
Other fully-featured, yet open-source, GPU-enabled MD packages include NAMD and GROMACS 4.6. Other projects include FenZi, but they don't seem to make their code available.
On the Joint Amber-Charmm (JAC) Benchmark, which consists of 23 558 atoms, but with a relatively short cutoff of 0.9nm, all these codes will get a handful of ns per day on a commodity GPU. That's still a few days of computing to get 1ms, but not bad considering that it's just one single machine.
Even though you have so much money available to spend on computing resources, the bigger issue, as Pedro points out, is that your problem is relatively small. With roughly 100,000 atoms, your "sweet spot" on CPU's will likely be about 100 cores. If you try to use more than that, you'll end up spending a lot of time communicating information between processors, which can be much more expensive. You could try to purchase multithreaded processors, but they might not help nearly as much.
However, you are relatively speaking in the sweet spot for GPU computing. So your best bet would be either to use a package like HOOMD, or to take advantage of shared-memory machines using the multithreaded package options in codes such as LAMMPS or NAMD to avoid some of the internal message passing.
For all-atom, explicit solvent, bio-molecular systems of O(100k) atoms you ought now to be using GPU-accelerated codes. Even without knowing exactly the setup of your simulations it is most probable that ACEMD, AMBER, Gromacs, NAMD would all be adequate for your needs.
Generally these codes won't scale beyond a single system for your simulation size (unless with network like Infiniband), or even a few GPUs, and strongly favour GPU performance over CPU, so focus on machine configurations with several high-performance GPUs and good PCIe connectivity. Plan for 1-2 core/GPU. With some codes, there's no need to have multi-CPU systems, since the computation is done on the GPU (note that GROMACS will use both CPU and GPU effectively, so their quality should be balanced), nor employ a high performance interconnect, such as Infiniband.
All the codes use CUDA so Nvidia GPUs are the way to go. Geforce cards are perfectly adequate (eg the 4GB Geforce GTX680), and substantially more economical than the Teslas.
We sell a workstation optimised for ACEMD and other MD codes Acellera Metrocubo. Alternatively, register for the NVidia GPU Test Drive to be put in touch with other suitable hardware resellers.
With regard to the criticism of hydrogen mass re-partitioning, the theoretical and technical basis was first described in:
Improving efficiency of large time-scale molecular dynamics simulations of hydrogen-rich systems Feenstra et al. JCC 1999
doi://10.1002/(SICI)1096-987X(199906)20:8<786::AID-JCC5>3.0.CO;2-B
It is a widely used method, implemented not only in ACEMD but also Gromacs and, recently, Anton:
Atomic-level description of ubiquitin folding, Piana et al, PNAS (2013)
doi://10.1073/pnas.1218321110
A deterministic code is not necessarily reversible, so it should not actually add any benefit in terms of statistical sampling, it kinds of produce the same errors but consistently. The consistency is useful for debugging. All codes are numerically integrated, so they will all drift sooner or later from the constant energy. It is important to stay close enough to the energy surface even when sampling in NVT and all codes used by people actually do it using mixed or fixed precision (NAMD, LAMMPS, ACEMD, AMBER, GROMACS, DESMOND, ANTON).
A single gtx680 would produce around 160 ns/day. A single gtx Titan 220 ns/day. So you get 1 microsecond in 4 days with the free version of acemd.
