17 votes
Accepted

Why aren't Krylov subspace methods popular in the Machine Learning community compared to Gradient Descent?

On a basic level, I don't buy the argument that you have to "solve a linear system for many machine learning algorithms". Much more, you usually have to optimize a non-linear equation which ...
user avatar
  • 2,477
5 votes

Why aren't Krylov subspace methods popular in the Machine Learning community compared to Gradient Descent?

They aren't popular because they don't work. Nicol N. Schraudolph spent a few years on Krylov-like methods for Machine Learning. I first learned of his work at 2004 Machine Learning Summer School in ...
user avatar
5 votes
Accepted

Notation for an optimization function that receives a vector of pairs

That seems correct, but I would suggest two minor improvements: Rearrange the pairs in a $(n+1)\times 2$ matrix $$ M = \begin{bmatrix} t_0 & v_0\\ \vdots & \vdots\\ t_n & v_n \end{bmatrix}...
user avatar
3 votes
Accepted

Optimization software for real-valued functions of complex arguments

I sat down and worked a bit on this. Since I am only interested in the magnitude of the polynomial it might be helpful to employ the polar representation, i.e., $1 - z_j \lambda = r_j e^{i\phi_j}$. ...
user avatar
  • 601
3 votes

How to ensure the numeric value is always positive in Optimization Python?

$ \def\bbR#1{{\mathbb R}^{#1}} \def\a{\alpha}\def\b{\beta}\def\l{\lambda} \def\o{{\tt1}}\def\p{\partial} \def\LR#1{\left(#1\right)} \def\diag#1{\operatorname{diag}\LR{#1}} \def\Diag#1{\operatorname{...
user avatar
  • 389
1 vote

Why aren't Krylov subspace methods popular in the Machine Learning community compared to Gradient Descent?

Posts 3 in the following series on "regret minimization" talks about gradient descent, and the later posts come back to that topic from time to time. Online Optimization Post 1: ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible