Multi-output ridge regression:

$$W^{*}=\underset{W}{\arg \min } \frac{1}{\mathcal{N}}\|Y-WX\|_{F}^{2}+\lambda\|W\|_{F}^{2}$$

There are $$Q$$ outputs, $$N$$ samples, and $$P$$ covariates (features).

$$\hat{Y}\in\mathbb R^{Q\times N}$$ is the prediction matrix,

$$\hat{Y}=WX$$

where $$W\in\mathbb R^{Q\times P}$$ is the parameter matrix. These are the parameters we estimate in the regression.

$$X\in\mathbb R^{P\times N}$$ is the data matrix.

I have been trying to implement a multi-output ridge regression algorithm. I have derived the gradient descent update for the parameter $$W$$.The evaluation metric I am using is RRMSE(Relative root mean square error). I have observed that my RRMSE values are going very high for certain values of hyperparameters $$\lambda$$ (L2-regularization constant) and $$\eta$$ which is the learning rate in gradient descent. My $$\eta$$ remains constant throughout the gradient descent procedure.

$$\operatorname{RRMSE}=\sqrt{\frac{\sum_{\left(\mathbf{x}_{i}, \mathbf{y}_{i}\right) \in D_{\text {test}}}\left(\hat{\mathbf{y}}_{i}-\mathbf{y}_{i}\right)^{2}}{\sum_{\left(\mathbf{x}_{i}, \mathbf{y}_{i}\right) \in D_{\text {test}}}\left(\hat{Y}-\mathbf{y}_{i}\right)^{2}}}$$

where $$\left(\mathbf{x}_{i}, \mathbf{y}_{i}\right)$$ is the $$i$$ th sample $$\mathbf{x}_{i}$$ with ground truth target $$\mathbf{y}_{i}, \hat{\mathbf{y}}_{i}$$ is the prediction of $$\mathbf{y}_{i}$$ and $$\hat{Y}$$ is the average of the targets over the training set $$D_{\text {train}} .$$ A lower RRMSE indicates better performance.

My attempts: I have read about the impact of un-normalized data on gradients but in my case the data is normalized.

I want some help with the two doubts I have right now.

$$1.$$What are the theoretically possible reasons for gradients to explode in gradient descent?

$$2.$$ Are there better ways to choose $$\eta$$ in order to avoid the exploding gradients problem.

• In geneal, a constant $\eta$ will only work if it is sufficiently small. – Brian Borchers Feb 14 at 5:01