I am running molecular dynamics (MD) simulations using several software packages, like Gromacs and DL_POLY.
Gromacs now supports both the particle decomposition and domain decomposition algorithms. By default, Gromacs simulations use domain decomposition, although for many years, until recently, particle decomposition was the only method implemented in Gromacs. In one of the Gromacs papers (DOI 10.1002/jcc.20291), the authors give a reason for their initial choice of particle decomposition:
"An early design decision was the choice to work with particle decomposition rather than domain decomposition to distribute work over the processors. In the latter case, spatial domains are assigned to processors, which enables finding spatial neighbors quickly by local communication only, but complications due to particles that move over spatial boundaries are considerable. Domain decomposition is a better choice only when linear system size considerably exceeds the range of interaction, which is seldom the case in molecular dynamics. With particle decomposition each processor computes the forces and coordinate/velocity updates for an assigned fraction of the particles, using a precomputed neighborlist evenly distributed over processors. The force $F_{ij}$ arising from the pair interaction between particles $i$ and $j$, which is needed for the velocity update of both particles $i$ and $j$, is computed only once and communicated to other processors. Every processor keeps in its local memory the complete coordinate set of the system rather than restricting storage to the coordinates it needs. This is simpler and saves communication overhead, while the memory claim is usually not a limiting factor at all, even for millions of particles. The neighborlist, on the other hand, which can contain up to 1000 times the number of particles, is distributed over the processors. Communication is essentially restricted to sending coordinates and forces once per time step around the processor ring. These choices have proven to be robust over time and easily applicable to modern processor clusters."
What do they mean by "linear system size" in the sentence "Domain decomposition is a better choice only when linear system size considerably exceeds the range of interaction, which is seldom the case in molecular dynamics"? From the paragraph above, I get the idea that particle decomposition has the advantage that one does not have to deal with particles moving across domain boundaries; rather, you just have to have enough memory for each processor to store the total system configuration. So particle decomposition is looking very favorable, whereas domain decomposition is looking very unfavorable.
I am sure that this is a very complicated question (and probably the subject of many books), but just basically, if particle decomposition seems so favorable, why would anyone need to use domain decomposition? Is domain decomposition just favorable if the system's size is very large (making it difficult or impossible to store the total configuration in each processor)? Based on the quoted paragraph above, I am not sure why domain decomposition is now, just recently, the default parallelization algorithm in Gromacs.
It seems that DL_POLY now (version 4) also uses domain decomposition. From the version 4 manual:
"The division of the conguration data in this way is based on the location of the atoms in the simulation cell, such a geometric allocation of system data is the hallmark of DD algorithms. Note that in order for this strategy to work efficiently, the simulated system must possess a reasonably uniform density, so that each processor is allocated almost an equal portion of atom data (as much as possible). Through this approach the forces computation and integration of the equations of motion are shared (reasonably) equally between processors and to a large extent can be computed independently on each processor. The method is conceptually simple though tricky to program and is particularly suited to large scale simulations, where efficiency is highest.
...
In the case of the DD strategy the SHAKE (RATTLE) algorithm is simpler than for the Replicated Data method of DL_POLY Classic), where global updates of the atom positions (merging and splicing) are required."
This makes it sound as if domain decomposition is good because it may be more efficient, even though perhaps more difficult to implement.
On the other hand, a previous version (DL_POLY Classic) used replicated data parallelization, which seems to be another name for particle decomposition. From that version's manual:
The Replicated Data (RD) strategy is one of several ways to achieve parallelisation in MD. Its name derives from the replication of the configuration data on each node of a parallel computer (i.e. the arrays defining the atomic coordinates $\textbf{r}_i$, velocities $\textbf{v}_i$, and forces $\textbf{f}_i$, for all $N$ atoms in the simulated system, are reproduced on every processing node). In this strategy most of the forces computation and integration of the equations of motion can be shared easily and equally between nodes and to a large extent be processed independently on each node. The method is relatively simple to program and is reasonably efficient. Moreover, it can be “collapsed” to run on a single processor very easily. However the strategy can be expensive in memory and have high communication overheads, but overall it has proven to be successful over a wide range of applications.
This paragraph seems generally consistant with the first paragraph in this question, except that it says that replicated data/particle decomposition has "high communication overheads." The paragraph from the Gromacs paper seems to say the opposite -- that particle decomposition is preferable because it has lower communication overhead than domain decomposition.
Do you have any thoughts?