In my optimization problem, the hessian has a structure such that it can be written as the sum of two matrices. Populating the first of the matrices is efficient. Populating the second one is computationally very heavy and practically makes methods like truncated Newton useless. However, the second matrix is block diagonal in nature. I am thinking I will use a Quasi-Newton approximation to the second matrix and compute the first matrix exactly. Hence, get an approximation for the hessian by adding these two matrices. Intuitively, this Hessian approximation should be better than having a Quasi-Newton approximation for the entire Hessian matrix. Is this thinking correct? Have people successfully tried such an approach before and got a better optimization approach than a direct Quasi-Newton based approach?
You may find this paper relevant. They give some BFGS-inspired methods and convergence proofs for cases where the Hessian is the sum of a part you can compute easily and a part you can't ("structured secant" methods), and you want to leverage your knowledge of the computable part as much as possible.