1
$\begingroup$

I'm comparing two optimization algorithms for deep neural networks. To that end, I train a network with the same data and starting rom the same initial weights using two different optimization algorithms. Then, I visualize the optimization trajectory within the loss landscape using this method.

Besides the qualitative observation that how trajectory moves w.r.t. the loss level-sets, are there any quantitive measures to compare the two methods?

enter image description here enter image description here

Improve this question
$\endgroup$
4
  • $\begingroup$ Number of functions evaluations needed to get to the minimum? Quality of the minimum found? (I.e., how far is the objective function value found by the algorithm from the true minimum? Similar for the parameter values, how far are those from the true one?). These are just a few “quantitative” measures of th goodness of an optimization algorithm $\endgroup$
    – Infinity77
    Jun 10, 2022 at 9:17
  • $\begingroup$ This is a classical problem: Comparing different optimization methods against each other for specific test cases. There are many papers that do this, and a number of benchmark collections that have been used for it. Search the literature and you will find :-) $\endgroup$
    – Wolfgang Bangerth
    Jun 11, 2022 at 22:21
  • $\begingroup$ @WolfgangBangerth Would you mind sharing one such examples that specifically looks at the loss landscapes? $\endgroup$
    – Blade
    Jun 11, 2022 at 23:16
  • $\begingroup$ @Blade I don't know the optimization literature well enough, though I suspect it won't be very difficult to find such comparison articles. $\endgroup$
    – Wolfgang Bangerth
    Jun 15, 2022 at 11:45

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy