The author of the SIAM News article "Optimization Theory and Perspectives on the Field of Machine Learning" mentions:

... For unconstrained convex optimization, GD (gradient descent) convergence rates do not depend on the space’s dimensionality. In other words, if we measure a notion of how close we are to the global minimum, convergence rates depend on the function’s properties and the number of descent steps; they are independent of whether a problem has 10 variables or 10 million. Readers might find this dimension-independence surprising, though perhaps there is a “catch” that practical problems may have a correlation between dimensionality and their Lipschitz constant.

I was wondering if there are any research papers/reports/studies that support the claim (in bold) that practical optimization problems have a correlation between dimensionality and their Lipschitz constant.

This may be a general study that performs an analysis on a large set of various problems or a narrow application with a dimensionality-Lipschitz constant study. It would be particularly helpful if the study is related to computational electromagnetics, but I am also interested in other domains. A more general optimization theory reference that would justify this [quite intuitive] claim (as optimization is not my main area of expertise) would be also quite welcome.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.