2
$\begingroup$

In what sense is the weighted Frobenius norm "adimensional"/"scale-invariant" for any symmetric positive definite weight matrix $W$? If we plug in a positive diagonal matrix into $W$ wee see that $||A||_W$ is not always $||TA||_W$ for a one-to-one transformation $T$.

See page 196 of Nocedal's book on numerical optimization and page 60 of Fletcher's "Practical methods of numerical optimization" for reference.

$\endgroup$
  • $\begingroup$ I don't think the claim is that this is the case for any weighting matrix, just for the one that actually gives rise to the BFGS update (and, possibly, not for any matrix A but just the ones obtained from BFGS updates). I think the claim is actually that the updates obtained from this norm are scale invariant (in the sense, e.g., of Deuflhard's "affine invariance"). $\endgroup$ – Christian Clason Jun 30 '16 at 21:22
  • 1
    $\begingroup$ Also also, please don't cross-post: math.stackexchange.com/q/1835397 (at least not without explicit mention). $\endgroup$ – Christian Clason Jun 30 '16 at 21:25
  • 1
    $\begingroup$ a) Don't trust claims without proof, not even in textbooks. Especially such vague statements. b) Textbooks tend to copy from each other, so no surprise there. c) The whole thing with the weighted norms is a post facto interpretation extracted from the convergence proofs and only meant to soften the blow of the various updates falling from the sky (which they did, historically) -- I am guilty of that myself when teaching nonlinear optimization. $\endgroup$ – Christian Clason Jun 30 '16 at 21:41
  • 1
    $\begingroup$ All this is to say that the only sensible interpretation of the statement is that these quasi-Newton updates have the same scale invariance (in the sense that scaling the function, doing a Newton step, and scaling back leads to the same iterates as doing a Newton step for the original function). I don't know of any proofs of the top of my head, though. Maybe this helps: people.eecs.berkeley.edu/~pabbeel/cs287-fa12/slides/… $\endgroup$ – Christian Clason Jun 30 '16 at 21:43
  • 1
    $\begingroup$ I did find this: ams.org/journals/mcom/1979-33-145/S0025-5718-1979-0514823-9/… (Section 7) and this: math.uh.edu/~rohop/fall_04/downloads/script.pdf (Section 4) $\endgroup$ – Christian Clason Jun 30 '16 at 21:50

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.