0
$\begingroup$

Multi-output ridge regression:

$$W^{*}=\underset{W}{\arg \min } \frac{1}{\mathcal{N}}\|Y-WX\|_{F}^{2}+\lambda\|W\|_{F}^{2}$$

There are $Q$ outputs, $N$ samples, and $P$ covariates (features).

$\hat{Y}\in\mathbb R^{Q\times N}$ is the prediction matrix,

$$\hat{Y}=WX$$

where $W\in\mathbb R^{Q\times P}$ is the parameter matrix. These are the parameters we estimate in the regression.

$X\in\mathbb R^{P\times N}$ is the data matrix.

I have been trying to implement a multi-output ridge regression algorithm. I have derived the gradient descent update for the parameter $W$.The evaluation metric I am using is RRMSE(Relative root mean square error). I have observed that my RRMSE values are going very high for certain values of hyperparameters $\lambda$ (L2-regularization constant) and $\eta$ which is the learning rate in gradient descent. My $\eta$ remains constant throughout the gradient descent procedure.

$$\operatorname{RRMSE}=\sqrt{\frac{\sum_{\left(\mathbf{x}_{i}, \mathbf{y}_{i}\right) \in D_{\text {test}}}\left(\hat{\mathbf{y}}_{i}-\mathbf{y}_{i}\right)^{2}}{\sum_{\left(\mathbf{x}_{i}, \mathbf{y}_{i}\right) \in D_{\text {test}}}\left(\hat{Y}-\mathbf{y}_{i}\right)^{2}}}$$

where $\left(\mathbf{x}_{i}, \mathbf{y}_{i}\right)$ is the $i$ th sample $\mathbf{x}_{i}$ with ground truth target $\mathbf{y}_{i}, \hat{\mathbf{y}}_{i}$ is the prediction of $\mathbf{y}_{i}$ and $\hat{Y}$ is the average of the targets over the training set $D_{\text {train}} .$ A lower RRMSE indicates better performance.

My attempts: I have read about the impact of un-normalized data on gradients but in my case the data is normalized.

I want some help with the two doubts I have right now.

$1.$What are the theoretically possible reasons for gradients to explode in gradient descent?

$2.$ Are there better ways to choose $\eta$ in order to avoid the exploding gradients problem.

Please provide your inputs on the above two doubts.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.