Does the loss function in a deep neural network act as a norm?

Question

I read somewhere that the Measn squared error loss function acts as L2 norm of the paramter vector. I would like to know if I am using binary cross entropy loss function, do I need to calculate the norm seperately? or the BCE loss function also acts as a norm? As an additional question I would like to know that is gradient calculation not enough for the back propagation or what is the role of the norm here?

Answer 1

Connection between Mean squared error and $L_2$ norm:

First the $L_2$ norm of a vector $\boldsymbol{x}\in\mathbb{R}^n$ is defined as

\begin{align*} \lVert \boldsymbol{x}\rVert_2=\left(\sum\limits_{i=1}^{n} x_i\right)^{1/2} \end{align*}

By the mean squared error (MSE) loss function, you want to measure the error between the prediction $\hat{\boldsymbol{y}}$ of your neural network and the actual target value $\boldsymbol{y}$. The MSE is defined as

\begin{align*} MSE(\hat{\boldsymbol{y}},\boldsymbol{y}):= \frac{1}{n}\sum\limits_{i=1}^{n} (\hat{{y}}_i-{y}_i)^2 =\frac{1}{n}\lVert \hat{y}-y\rVert_2^2 \end{align*}

You see at the right that the MSE can be rewritten in terms of the $L_2$ norm from above. What you should realize here is that norms can be used to measure the difference between objects in vector spaces.

Now to your question:

I would like to know if I am using binary cross entropy loss function, do I need to calculate the norm separately?

This can be answered with yes and no, depending on what you actually want to do. To be more clear, you can train your neural network by using only the BCE loss. But you are likely to end up with a network that overfits your data. Have a look at the bias variace tradeoff.

For example assume that you have a neural network $f(\boldsymbol{x},\theta)$ with input-features $\boldsymbol{x}$ and parameters $\theta$. Moreover, assume you use some loss function $\ell(\hat{\boldsymbol{y}},\boldsymbol{y})$. Then to find $\theta$ you usually try to solve the empirical risc minimization problem on the training set $S_{train}=\left\{(\boldsymbol{x}_1,\boldsymbol{y}_1),...,(\boldsymbol{x}_N,\boldsymbol{y}_N)\right\}$ by

\begin{align*} \min_{\theta}\sum\limits_{j=1}^{N}\frac{1}{N}\ell(f(\boldsymbol{x}_j,\theta),\boldsymbol{y}_j) \end{align*}

So if you choose $\ell$ as BCE loss you just calculate the gradient $\nabla_\theta\sum\limits_{j=1}^{N}\ell(f(\boldsymbol{x}_j,\theta),\boldsymbol{y}_j)$ and use it for the gradient descent update.

You can in-cooperate norm regularization to prevent your neural network from overfitting. For that, you change the minimization problem in the following way:

\begin{align*} \min_{\theta}\left(\sum\limits_{j=1}^{N}\frac{1}{N}\ell(f(\boldsymbol{x}_j,\theta),\boldsymbol{y}_j)+ \gamma||\theta||_2\right) \end{align*}

where $\gamma$ is a scalar. And then again take the gradient $\nabla_\theta$ when using it in the gradient descent method. Since we minimize the expression above, the additional $\gamma||\theta||_2$ is forced to be low. By making $\gamma$ large, you can put more pressure on this constraint.

If you want to know more about norm regularization for neural networks, check out Chapter 7 in the deep learning book.