# Confusion about matrix differentiation in a nonlinear matrix equation

I am trying to solve a matrix equation in the following discrete form: $$\frac{K^{n+1}-K^n}{\Delta t} = [(K^{n+1} (V^{n})^T).^3 - K^{n+1} (V^{n})^T]V^n.$$ where $$K^{n+1} \in \mathbb{R}^{m \times r}, V \in \mathbb{R}^{n \times r}$$, $$V^T V = I$$(identity matrix) and $$.^3$$ are elementwise operations.

I am attempting to solve using Newton iterate method, which means I need to differentiate the function F： $$F = K^{n+1}-K^n-\Delta t[(K^{n+1} (V^{n})^T).^3 - K^{n+1} (V^{n})^T]V^n.$$ I differentiate F with respect to $$K^{n+1}$$ and get: $$\frac{\partial F}{ \partial K} = I_K - \Delta t [3(K^{n+1} (V^{n})^T).^2 (V^{n})^T - (V^{n})^T]V^n.$$ In my understanding, if a matrix is differentiated with respect to itself, the result should be an identity matrix. However, since $$K \in \mathbb{R}^{m \times r}$$ is not a square matrix, how should this be handled? How can I solve the matrix equation? Thanks in advance.

• The directional derivative of $A^3$ in direction $B$ is $A^2B+ABA+BA^2$, there is no further reduction as the matrix multiplication is not commutative. Commented Mar 1 at 7:01
• @LutzLehmann That was my first reaction, too, but then I noticed that those in OP's question are elementwise products, not matrix products. Commented Mar 1 at 8:01
• Ok, if that is indeed a compact form to write the discretized version of a PDE, then the factor 3 is correct. But then also, as per the answer, the identity matrix is not correct. Commented Mar 1 at 8:35

If a matrix is differentiated with respect to itself, the result should be a fourth order tensor. The easist way to see this is to work with components.

$$\frac{ \partial K_{ij}}{\partial {K_{kl}}} = \delta_{ik}\delta_{jl}$$

$$\delta_{ij}$$ is the Kronecker delta. Because differentiation survives only when $$i=k$$ and $$j=k$$. If your matrix is symmetric, then

$$\frac{ \partial K_{ij}}{\partial {K_{kl}}} = \delta_{ik}\delta_{jl} + \delta_{il}\delta_{jk}$$

Because, the differentiation also survives for $$i=l$$ and $$j=k$$.

Edit:

The correct answer for a symmetric matrix is

$$\frac{ \partial K_{ij}}{\partial {K_{kl}}} = \frac{1}{2} (\delta_{ik}\delta_{jl} + \delta_{il}\delta_{jk} )$$

See this

• Thank you for your answer. If $K \in \mathbb{R}^{m \times n}$, then the tensor corresponding to $\partial K$ is of size $m \times n \times m \times n$, is that correct? Commented Mar 1 at 5:04
• Yes, I think that's correct, that's the standard ordering for $\frac{ \partial K_{ij}}{\partial {K_{kl}}}$
– NNN
Commented Mar 1 at 5:41
• If you are just interested in the directional derivative in a direction $H$, though, i.e., $\frac{d}{dt} F(K+tH)$ that's an $m\times n$ matrix (which is a function of $H$). Commented Mar 1 at 11:37
• I'm pretty sure I'm missing a factor of 1/2 in the symmetric case.
– NNN
Commented Mar 2 at 7:01
• @FedericoPoloni I would be grateful for a better explanation of the symmetric matrix derivative wrt itself, than the one I linked to.
– NNN
Commented Mar 2 at 10:49

$$\def\R#1{{\mathbb R}^{#1}} \def\o{{\large\tt1}} \def\D{{\cal D}} \def\k{\otimes} \def\h{\odot} \def\bR#1{\big(#1\big)} \def\BR#1{\Big(#1\Big)} \def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\vc#1{\op{vec}\LR{#1}} \def\Diag#1{\op{Diag}\LR{#1}} \def\qiq{\quad\implies\quad} \def\mt{\mapsto} \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\c#1{\color{red}{#1}} \def\CLR#1{\c{\LR{#1}}}$$First let's clean up some variable names by stripping off unnecessary decorations \eqalign{ \LR{\Delta t,V^T,K^{n+1},K^n} \;\mt\; \LR{h,A,X,Y} \\ } Let's also use standard notation for elementwise products and powers \eqalign{ X^{\h 3} = X\h X\h X \\ } And introduce the all-ones matrix $$\o\in\R{m\times n}$$ and identity matrix $$I_m\in\R{m\times m}$$

Rewrite the function and calculate its differential \eqalign{ F &= X - Y - hI_m\BR{\LR{XA}^{\h 3} - XA}A^T \\ dF &= dX - hI_m\BR{\LR{3\LR{XA}^{\h 2} - \o}\odot\bR{dX\,A}}A^T \\ } To avoid introducing tensors, vectorize the equation and calculate the Jacobian \eqalign{ df &= \vc{dF} \\ &= dx-h\LR{A\k I_m}\,\vc{\LR{3\LR{XA}^{\h 2}-\o}\odot\bR{dX\,A}} \\ &= dx-h\LR{A\k I_m}\,\c{\Diag{\vc{3\LR{XA}^{\h 2}-\o}}}\,\vc{dX\,A} \\ &= dx - h\LR{A\k I_m}\,\c{\D}\,\LR{A^T\k I_m}dx \\ J\doteq\grad fx &= I_{mr} - h\LR{A\k I_m}\,\c{\D}\,\LR{A^T\k I_m} \;\in\; \R{mr\times mr} } NB: The Jacobian is a symmetric matrix.

Newton's method requires calculating $$J^{-1}\,$$ at every step, which is a standard operation for a matrix, but what does that mean for a fourth-order tensor?

IOW, you could calculate an expression for $$\large\grad FX,\,$$ but what would you do with it?