There are some great libraries with linear solvers for sparse matrices - SuiteSparse is the obvious one. The methods work on sparse matrices with scalar entries.
However, often in optimization problems, the matrices consist of dense sub-blocks within a sparse structure. Are there any solver libraries that define block sparse matrix structures, and have solvers that take advantage of the structure?
In particular, I have a system of normal equations from bundle adjustment, which generates a symmetric sparse matrix with some blocks of side-length 9 and and others of side-length 3. I'd like to use the CHOLMOD algorithm to solve this, but exploiting the block structure. Do any libraries do this?