The derivation of the BFGS is more intuitive when one considers (strictly) convex cost functionals:
However, some
background information is necessary: Assume, one wants to minimize a convex functional
$$
f(x) \to \min_{x\in \mathbb R^n}.
$$
Say there is an approximate solution $x_k$. Then, one approximates the minimum of $f$ by the minimum of the truncated Taylor expansion
$$
f(x_k+p) \approx f(x_k) +\nabla f(x_k)^Tp + \frac{1}{2}p^T H(x_k)p. \quad(*)
$$
That is, one looks for $p$ such that $(*)$ is minimal and sets $x_{k+1} := x_k + p$. Computing the gradient of $(*)$ -- "with respect to $p$" -- and setting it to zero gives the relation
$$
H(x_k)[x_{k+1} - x_k] = \nabla f(x_{k+1}) - \nabla f(x_k),
$$
where $H$ is the 'Jacobian of the gradient' or the Hessian matrix.
Since the computation and inversion of the Hessian is expensive...
...a short answer
(cf. Broyden's update) might be that the BFGS update $H_{k+1}^{-1}$ minimizes
$$
\|H_k^{-1} - H^{-1}\|_W
$$
in a smartly chosen weighted Frobenius norm, subject to
- $H[x_{k+1} - x_k] = \nabla f(x_{k+1}) - \nabla f(x_k)$ -- this is what one is out for -- and
- $H^T = H$, because the Hessian is symmetric.
Then the choice of the weight $W$ in $\|H\|_W := \|W^{1/2}HW^{1/2}\|_F$
as the inverse of the averaged Hessian $G:=\int_0^1 H(x_k + \tau p) d\tau$, cf. here for the statement but without proof, gives the BFGS update formula (with $\alpha_k = 1$).
The major points are:
- One tries to approximate the solution for the actual costs by the solution for a quadratic approximation
- Computation of the Hessian, and its inverse, is expensive. One prefers simple updates.
- The update is chosen optimal for the inverse rather than the actual Hessian.
- That it is a rank-2 update is a consequence of the particular choice of the weights in the Frobenius norm.
A longer answer, should include how to choose the weights, how to make this work for nonconvex problems (where a curvature condition appears that requires a scaling of the search direction $p$), and how to derive actual the formula for the update. A reference is here (in German).