Effect of Initial guess B (approximate Hessian) on BFGS algorithm

I am trying to implement BFGS. The purpose is to approximate Hessian matrix only (not using the quasi-newton optimization steps), so i am using steepest ascent for optimization. What I observe is that the final Hessian approximate is very sensitive to the initial guess of Hessian. If I start with Identity matrix, most of the singular values of final hessian is closed to 1. Similarly if I start with any multiple of identity (let's say 5*I), the singular values of final Hessian is closed to 5.

It is a maximization problem, and also the solution is not unique so i expect a negative semi-definite Hessian matrix.

I hope my question is clear.

There are two standard choices for the initial approximation of $B$ in BFGS: Either you choose $B=\frac{\|g_0\|}{\delta} I$ where $\|g_0\|$ is the gradient in the very first iteration and $\delta$ a "typical step size" from $x_k$ to $x_{k+1}$. Or you choose $B=\frac{y_1^T y_1}{y_1^T s_1}I$ using the standard notation used for BFGS.
• If you do a few hundred BFGS iterations (in particular, many times the number of variables), then the effect of the initial guess for $B$ may have disappeared, but if you only do a few iterations, then the initial guess will still be prominent. Apr 10, 2014 at 12:59
• What I mean is that if you apply BFGS to a minimization problem and you choose the step length appropriately, the matrix $B$ will always be positive definite even if the Hessian $H$ is not. Nocedal and Wright have much more material on this, I would recommend taking a look at their book. Apr 11, 2014 at 12:30