There are many different linear solvers, some which work best for diagonally dominant matrices, some for symmetric, some for positive definite ones, some for banded matrices, etc... There are direct methods, iterative methods, Krylov space methods, etc... Is there a good heuristic to choose a linear solver that scales well in parallel as the problem size increases? In particular, is there some sort of block diagram that can specify an efficient (or optimal) solver for specific matrix system types?
I presume that by "optimal" you mean "fastest solve".
I think the PDE solver guys have the best advice when it comes to advice about the most appropriate algorithm for certain discretizations and problems, such as Helmholtz problems (see, for instance, this question) and other cases where problem structure can be exploited.
From a basic numerical linear algebra perspective, here are some general guidelines:
If your problem is dense, not low-rank, and small enough to fit into memory, direct solvers are a decent choice.
If your problem is ill-conditioned, preconditioning or regularizing is a good idea.
If you can exploit structure (bandedness, symmetry, diagonal dominance), doing so will save you operations.
If your problem is sparse and small enough, sparse direct solvers can work well.
If your problem is sparse and large, use an iterative solver, because fill-in will render sparse direct solvers useless. The efficacy of an iterative solver is problem-dependent (see this question) and depends on spectrum. You will probably want to try all of them, unless your problem has a specific structure, or you know from experience that a certain algorithm works well for your problem.
If your problem is dense, and low-rank, you may be able to exploit that low-rank structure, either through use of iterative solvers, or specialized algorithms (which require special problem structure, as alluded to above with PDEs).
When it comes to parallelism, the best algorithms minimize communication, which is accomplished by structuring both the algorithm and the data accordingly. I am not an expert in these matters. I know that block cyclic matrix decompositions are popular for dense numerical linear algebra, and...that's it. I work almost entirely on serial algorithms, so I leave the discussion of parallelism to others.
It's probably not even close to optimal, but here's the basic hueristic I use:
Don't use a direct solver unless the matrix involved is diagonal, tridiagonal, can be easily put into tridiagonal form (as with the split operator method), or has some other property that makes it easy to directly solve.
Use whatever iterative solver your favorite library uses otherwise. Iterative solvers tend to be nicer to parallelize.
Note that there are lots of specific cases where you really need some specific algorithm, as indicated by this answer.