Multi-output ridge regression:

$$W^{*}=\underset{W}{\arg \min } \frac{1}{\mathcal{N}}\|Y-WX\|_{F}^{2}+\lambda\|W\|_{F}^{2}$$

There are $Q$ outputs, $N$ samples, and $P$ covariates (features).

$\hat{Y}\in\mathbb R^{Q\times N}$ is the prediction matrix,


where $W\in\mathbb R^{Q\times P}$ is the parameter matrix. These are the parameters we estimate in the regression.

$X\in\mathbb R^{P\times N}$ is the data matrix.

I have been trying to implement a multi-output ridge regression algorithm. I have derived the gradient descent update for the parameter $W$.The evaluation metric I am using is RRMSE(Relative root mean square error). I have observed that my RRMSE values are going very high for certain values of hyperparameters $\lambda$ (L2-regularization constant) and $\eta$ which is the learning rate in gradient descent. My $\eta$ remains constant throughout the gradient descent procedure.

$$\operatorname{RRMSE}=\sqrt{\frac{\sum_{\left(\mathbf{x}_{i}, \mathbf{y}_{i}\right) \in D_{\text {test}}}\left(\hat{\mathbf{y}}_{i}-\mathbf{y}_{i}\right)^{2}}{\sum_{\left(\mathbf{x}_{i}, \mathbf{y}_{i}\right) \in D_{\text {test}}}\left(\hat{Y}-\mathbf{y}_{i}\right)^{2}}}$$

where $\left(\mathbf{x}_{i}, \mathbf{y}_{i}\right)$ is the $i$ th sample $\mathbf{x}_{i}$ with ground truth target $\mathbf{y}_{i}, \hat{\mathbf{y}}_{i}$ is the prediction of $\mathbf{y}_{i}$ and $\hat{Y}$ is the average of the targets over the training set $D_{\text {train}} .$ A lower RRMSE indicates better performance.

My attempts: I have read about the impact of un-normalized data on gradients but in my case the data is normalized.

I want some help with the two doubts I have right now.

$1.$What are the theoretically possible reasons for gradients to explode in gradient descent?

$2.$ Are there better ways to choose $\eta$ in order to avoid the exploding gradients problem.

Please provide your inputs on the above two doubts.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.