I keep seeing talks in which people "rediscover" it and proceed to write down an algorithm that is algebraically-equilavent to one form or another of an existing gradient descent or Newton-like method, and fail to cite anyone else. I think it's not very useful because the conclusion is basically that "as long as you take small enough steps, the method eventually converges to a local minimum". Well, 2014 marks the 45th anniversary of Philip Wolfe's paper showing how to do this in a principled way. There is also good theory for obtaining q-quadratic or q-superlinear convergence from pseudotransient continuation and related methods like Levenberg-Marquardt.

If you want an instance of this rediscovery using a Newton-like formulation for solving algebraic equations (i.e., classical pseudotransient continuation) from a mathematician with more than 600 papers (so maybe he'll prove things you find interesting), look at the "Dynamical Systems Method" by A.G. Ramm [1].

If the intuition gained by considering a transient system led to practical algorithms that were either faster or more reliable, I think we'd see highly-cited articles on that subject. I think it's no mystery that Nocedal and Wright has over 13000 citations while Ramm's book has about 80 (mostly self-citations).

[1] I can advise you not to inform Prof. Ramm that his DSM is algebraically-equivalent to something that has been in countless engineering packages for decades or you may get yourself yelled out of the room. #gradstudentmemories

While I haven't seen the exact formulation that you have written down here, I keep seeing talks in which people "rediscover" a connection to integrating some transient system, and proceed to write down an algorithm that is algebraically-equilavent to one form or another of an existing gradient descent or Newton-like method, and fail to cite anyone else. I think it's not very useful because the conclusion is basically that "as long as you take small enough steps, the method eventually converges to a local minimum". Well, 2014 marks the 45th anniversary of Philip Wolfe's paper showing how to do this in a principled way. There is also good theory for obtaining q-quadratic or q-superlinear convergence from pseudotransient continuation and related methods like Levenberg-Marquardt.

If you want an instance of this rediscovery using a Newton-like formulation for solving algebraic equations (i.e., classical pseudotransient continuation) from a mathematician with more than 600 papers (so maybe he'll prove things you find interesting), look at the "Dynamical Systems Method" by A.G. Ramm [1].

If the intuition gained by considering a transient system led to practical algorithms that were either faster or more reliable, I think we'd see highly-cited articles on that subject. I think it's no mystery that Nocedal and Wright has over 13000 citations while Ramm's book has about 80 (mostly self-citations).

[1] I can advise you not to inform Prof. Ramm that his DSM is algebraically-equivalent to something that has been in countless engineering packages for decades or you may get yourself yelled out of the room. #gradstudentmemories