I am solving a multiscale problem using the Heterogeneous Multiscale Method (HMM). Essentially, my particular procedure uses the following iterative process:
- Solve many local matrix systems.
- Compute a value of interest from the solutions of the local systems.
- Assemble a global matrix system from the local "values of interest"
- Solve the global matrix system
- Use the solution of the global matrix system to form new local matrix systems.
Repeat until some convergence criteria is met.
Since there are many local (independent) linear systems of equations and multiple systems can fit into local RAM memory, I figure it is best to load multiple "local" systems into each processor and solve each system sequentially (see this posted question).
My question regards the best strategy to assemble and solve the global matrix system. In my particular case, the global matrix system is small enough that it could fit entirely on any processor's RAM memory. Furthermore, the local & global matrices do not change size between iterations. So, I foresee one of three possible strategies:
- Gather the "values of interest" onto a single processor, and assemble/solve the global matrix system sequentially on one processor.
- Copy the values of interest onto every processor, and assemble/solve the same global matrix system sequentially on every processor.
- Assuming that each processor possesses the "values of interest" necessary to produce contiguous blocks of the global matrix, then we can assemble partitions of the global matrix locally, then solve them together in parallel.
I can see some advantages/disadvantages to each method. In Method 1, no communication is necessary in the solving phase, but communication to and from the root processor may become a bottleneck (especially at scale). Method 2 may require more interprocessor communications to assemble the global matrix than the first method, but no communication is needed in the solving phase or in the local matrix assembly stage that follows. Method 3 requires no interprocessor communication for assembly of the local or global matrices, but requires it in the solving phase.
Suppose that each local system is on the order of $10^3$x$10^3$ and there are $10^3$x$10^3$ local matrix systems. Let's further suppose that the global matrix system has size $10^3$x$10^3$. Under these assumptions, which one the three aforementioned strategies will likely lead to a faster solution of the global system? Are there other mapping strategies for the global matrix that might work faster per iteration?