The row loss gradients

Question

Suppose the original loss function is $$\min_{\mathbf{V}}\frac{1}{2}\|\mathbf{V} - Q(\mathbf{V})\odot\mathbf{U}\mathbf{E} - \beta Q(\mathbf{V})\mathbf{V}\|_2^2$$ where $\odot$ denotes the element-wise product, $\mathbf{V}$ is a n by 1 vector, $\mathbf{U}$ is a n by n matrix, $\mathbf{E}$ is a all-ones vector, $Q$ is combinatorial matrix affected by $\mathbf{V}$, and $\beta$ is a scalar, therefore the loss function is a norm of a vector. I note that the minimization of this loss can be re-cast to the minimization of each row of the vector, i.e., minimize the row loss: $$\min_\mathbf{V} \frac{1}{2}\left(e_i^\top\mathbf{V} - Q_i^\top\mathbf{U}_i - \beta Q_i^\top \mathbf{V}\right)^2$$ where $e_i$ is a standard basis and the i-th element is 1. Now the row loss gradients is given by $$\nabla = [e_i^\top\mathbf{V}-Q_i^\top\mathbf{U}_i - \beta Q_i^\top](e_i - \beta Q_i)$$

Questions:

(1) Is the row loss gradients an alternative to the original loss gradients? Is it feasible to reformulate the original loss to row loss?

(2) Is there any math property between these two loss gradients?

Answer 1

Introduce the following variables $$\eqalign{ \def\h{\tfrac12} \def\o{{\tt1}} \def\p{\partial} \def\U{{\cal U}} \def\b{\beta} \def\qiq{\quad\iff\quad} \def\grad#1#2{\frac{\p #1}{\p #2}} \U &= {\rm Diag}({\rm vec}(U)) \\ q &= {\rm vec}(Q) \\ dq &= J\,dv \qiq J = \grad{q}{v} \\ x &= (I-\b Q)\,v - (U\odot Q)\,\o \\ dx &= (I-\b Q)\,dv - \b\:dQ\,v - (U\odot dQ)\o \\ &= (I-\b Q)\,dv - (\b v\otimes I)^Tdq - (\o\otimes I)^T\U\:dq \\ &= (I-\b Q)\,dv - (\b v\otimes I)^TJ\,dv - (\o\otimes I)^T\U J\,dv \\ &= K\,dv \\ }$$ and the Frobenius product $$\eqalign{ A:B &= \sum_{i=1}^m\sum_{j=1}^n A_{ij}B_{ij} \;=\; {\rm trace}(A^TB) \\ A:A &= \|A\|_F^2 \qquad \{ {\rm Frobenius\;norm} \} \\ }$$ The terms of a Frobenius product can be rearranged in many useful ways $$\eqalign{ A:B &= B:A = B^T:A^T \\ (AB):C &= A:(CB^T) = B:(A^TC) \\ }$$

---

Use the above notation to rewrite the original loss function and calculate its gradient $$\eqalign{ \Phi &= \h(x:x) \\ d\Phi &= \h(x:dx + dx:x) \\&= x:dx \\&= x:(K\,dv) \\&= (K^Tx):dv \\ \grad{\Phi}{v} &= K^Tx \\ }$$ What you've called the row loss function is $$\eqalign{ \phi_i &= \h(x_i:x_i) \;=\; \frac{x_i^2}{2} \\ }$$ which is related to the original loss function by $$\eqalign{ \def\qiq{\quad\implies\quad} \Phi &= \sum_{i=1}^n \:\phi_i \qiq \grad{\Phi}{v} &= \sum_{i=1}^n \:\grad{\phi_i}{v} \\ }$$