0
$\begingroup$

Suppose the original loss function is $$\min_{\mathbf{V}}\frac{1}{2}\|\mathbf{V} - Q(\mathbf{V})\odot\mathbf{U}\mathbf{E} - \beta Q(\mathbf{V})\mathbf{V}\|_2^2$$ where $\odot$ denotes the element-wise product, $\mathbf{V}$ is a n by 1 vector, $\mathbf{U}$ is a n by n matrix, $\mathbf{E}$ is a all-ones vector, $Q$ is combinatorial matrix affected by $\mathbf{V}$, and $\beta$ is a scalar, therefore the loss function is a norm of a vector. I note that the minimization of this loss can be re-cast to the minimization of each row of the vector, i.e., minimize the row loss: $$\min_\mathbf{V} \frac{1}{2}\left(e_i^\top\mathbf{V} - Q_i^\top\mathbf{U}_i - \beta Q_i^\top \mathbf{V}\right)^2$$ where $e_i$ is a standard basis and the i-th element is 1. Now the row loss gradients is given by $$\nabla = [e_i^\top\mathbf{V}-Q_i^\top\mathbf{U}_i - \beta Q_i^\top](e_i - \beta Q_i)$$

Questions:

(1) Is the row loss gradients an alternative to the original loss gradients? Is it feasible to reformulate the original loss to row loss?

(2) Is there any math property between these two loss gradients?

$\endgroup$

1 Answer 1

2
$\begingroup$

Introduce the following variables $$\eqalign{ \def\h{\tfrac12} \def\o{{\tt1}} \def\p{\partial} \def\U{{\cal U}} \def\b{\beta} \def\qiq{\quad\iff\quad} \def\grad#1#2{\frac{\p #1}{\p #2}} \U &= {\rm Diag}({\rm vec}(U)) \\ q &= {\rm vec}(Q) \\ dq &= J\,dv \qiq J = \grad{q}{v} \\ x &= (I-\b Q)\,v - (U\odot Q)\,\o \\ dx &= (I-\b Q)\,dv - \b\:dQ\,v - (U\odot dQ)\o \\ &= (I-\b Q)\,dv - (\b v\otimes I)^Tdq - (\o\otimes I)^T\U\:dq \\ &= (I-\b Q)\,dv - (\b v\otimes I)^TJ\,dv - (\o\otimes I)^T\U J\,dv \\ &= K\,dv \\ }$$ and the Frobenius product $$\eqalign{ A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; {\rm trace}(A^TB) \\ A:A &= \|A\|_F^2 \qquad \{ {\rm Frobenius\;norm} \} \\ }$$ The terms of a Frobenius product can be rearranged in many useful ways $$\eqalign{ A:B &= B:A = B^T:A^T \\ (AB):C &= A:(CB^T) = B:(A^TC) \\ }$$


Use the above notation to rewrite the original loss function and calculate its gradient $$\eqalign{ \Phi &= \h(x:x) \\ d\Phi &= \h(x:dx + dx:x) \\&= x:dx \\&= x:(K\,dv) \\&= (K^Tx):dv \\ \grad{\Phi}{v} &= K^Tx \\ }$$ What you've called the row loss function is $$\eqalign{ \phi_i &= \h(x_i:x_i) \;=\; \frac{x_i^2}{2} \\ }$$ which is related to the original loss function by $$\eqalign{ \def\qiq{\quad\implies\quad} \Phi &= \sum_{i=1}^n \:\phi_i \qiq \grad{\Phi}{v} &= \sum_{i=1}^n \:\grad{\phi_i}{v} \\ }$$

$\endgroup$
2
  • $\begingroup$ Thank you Greg, I am more than happy to take this as an accepted answer. So can I interpret this as the whole loss gradients are the summation of the row loss gradients? $\endgroup$ Commented Apr 25, 2023 at 9:02
  • 1
    $\begingroup$ Yes, that is correct. $\endgroup$
    – greg
    Commented Apr 25, 2023 at 11:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.