I need to solve $Ax=b$ in about 1 million dimensions. Furthermore, A is only accessible through matrix vector products, and these are noisy/inexact.

Is there any solver with Python interface I can try on this problem?

Basically I want to check if there's anything in the linalg community that might work to train a neural network better than current method. The current method is SGD which is equivalent to linearizing the problem and applying first-order Richardson iteration.

  • $\begingroup$ Neural network training generally involves minimizing a loss function that is nonlinear due to the activation functions that are being used. Are you actually looking to solve a large linear system of $n$ equations in $n$ unknowns? Or, a linear least squares problem with more equations than unknowns? Or, a nonlinear least squares problem? $\endgroup$ Oct 8 '20 at 4:01
  • $\begingroup$ Correct, it is a non-linear system, but the current approach used by everyone is to repeatedly linearize this system and apply first-order Richardson iteration. So I want to see if I can plug in another solver and get better results $\endgroup$ Oct 8 '20 at 17:13
  • $\begingroup$ There's a very large literature on optimization with inexact gradients and applications to neural network training. The methods are very unlike the methods used to solve large scale linear systems of equations arising from the numerical solution of PDE's where precise vector-matrix multiplications are available. $\endgroup$ Oct 8 '20 at 17:43
  • $\begingroup$ I'm pretty familiar with the neural network side of this literature (I've been a reviewer for NIPS+ICML+ICLR for the last 10 years). I'm more curious about taking an implementation from numerical linalg community and seeing how well it performs $\endgroup$ Oct 8 '20 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.