I am trying to solve a matrix equation in the following discrete form: $$ \frac{K^{n+1}-K^n}{\Delta t} = [(K^{n+1} (V^{n})^T).^3 - K^{n+1} (V^{n})^T]V^n. $$ where $K^{n+1} \in \mathbb{R}^{m \times r}, V \in \mathbb{R}^{n \times r}$, $V^T V = I$(identity matrix) and $.^3$ are elementwise operations.

I am attempting to solve using Newton iterate method, which means I need to differentiate the function F: $$ F = K^{n+1}-K^n-\Delta t[(K^{n+1} (V^{n})^T).^3 - K^{n+1} (V^{n})^T]V^n. $$ I differentiate F with respect to $K^{n+1}$ and get: $$ \frac{\partial F}{ \partial K} = I_K - \Delta t [3(K^{n+1} (V^{n})^T).^2 (V^{n})^T - (V^{n})^T]V^n. $$ In my understanding, if a matrix is differentiated with respect to itself, the result should be an identity matrix. However, since $K \in \mathbb{R}^{m \times r}$ is not a square matrix, how should this be handled? How can I solve the matrix equation? Thanks in advance.

  • 1
    $\begingroup$ The directional derivative of $A^3$ in direction $B$ is $A^2B+ABA+BA^2$, there is no further reduction as the matrix multiplication is not commutative. $\endgroup$ Commented Mar 1 at 7:01
  • $\begingroup$ @LutzLehmann That was my first reaction, too, but then I noticed that those in OP's question are elementwise products, not matrix products. $\endgroup$ Commented Mar 1 at 8:01
  • $\begingroup$ Ok, if that is indeed a compact form to write the discretized version of a PDE, then the factor 3 is correct. But then also, as per the answer, the identity matrix is not correct. $\endgroup$ Commented Mar 1 at 8:35

2 Answers 2


If a matrix is differentiated with respect to itself, the result should be a fourth order tensor. The easist way to see this is to work with components.

$$ \frac{ \partial K_{ij}}{\partial {K_{kl}}} = \delta_{ik}\delta_{jl} $$

$\delta_{ij}$ is the Kronecker delta. Because differentiation survives only when $i=k$ and $j=k$. If your matrix is symmetric, then

$$ \frac{ \partial K_{ij}}{\partial {K_{kl}}} = \delta_{ik}\delta_{jl} + \delta_{il}\delta_{jk} $$

Because, the differentiation also survives for $i=l$ and $j=k$.


The correct answer for a symmetric matrix is

$$ \frac{ \partial K_{ij}}{\partial {K_{kl}}} = \frac{1}{2} (\delta_{ik}\delta_{jl} + \delta_{il}\delta_{jk} ) $$

See this

  • $\begingroup$ Thank you for your answer. If $K \in \mathbb{R}^{m \times n}$, then the tensor corresponding to $\partial K$ is of size $m \times n \times m \times n$, is that correct? $\endgroup$
    – Owen Jun
    Commented Mar 1 at 5:04
  • $\begingroup$ Yes, I think that's correct, that's the standard ordering for $\frac{ \partial K_{ij}}{\partial {K_{kl}}}$ $\endgroup$
    – NNN
    Commented Mar 1 at 5:41
  • 1
    $\begingroup$ If you are just interested in the directional derivative in a direction $H$, though, i.e., $\frac{d}{dt} F(K+tH)$ that's an $m\times n$ matrix (which is a function of $H$). $\endgroup$ Commented Mar 1 at 11:37
  • $\begingroup$ I'm pretty sure I'm missing a factor of 1/2 in the symmetric case. $\endgroup$
    – NNN
    Commented Mar 2 at 7:01
  • $\begingroup$ @FedericoPoloni I would be grateful for a better explanation of the symmetric matrix derivative wrt itself, than the one I linked to. $\endgroup$
    – NNN
    Commented Mar 2 at 10:49

$ \def\R#1{{\mathbb R}^{#1}} \def\o{{\large\tt1}} \def\D{{\cal D}} \def\k{\otimes} \def\h{\odot} \def\bR#1{\big(#1\big)} \def\BR#1{\Big(#1\Big)} \def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\vc#1{\op{vec}\LR{#1}} \def\Diag#1{\op{Diag}\LR{#1}} \def\qiq{\quad\implies\quad} \def\mt{\mapsto} \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\c#1{\color{red}{#1}} \def\CLR#1{\c{\LR{#1}}} $First let's clean up some variable names by stripping off unnecessary decorations $$\eqalign{ \LR{\Delta t,V^T,K^{n+1},K^n} \;\mt\; \LR{h,A,X,Y} \\ }$$ Let's also use standard notation for elementwise products and powers $$\eqalign{ X^{\h 3} = X\h X\h X \\ }$$ And introduce the all-ones matrix $\o\in\R{m\times n}$ and identity matrix $I_m\in\R{m\times m}$

Rewrite the function and calculate its differential $$\eqalign{ F &= X - Y - hI_m\BR{\LR{XA}^{\h 3} - XA}A^T \\ dF &= dX - hI_m\BR{\LR{3\LR{XA}^{\h 2} - \o}\odot\bR{dX\,A}}A^T \\ }$$ To avoid introducing tensors, vectorize the equation and calculate the Jacobian $$\eqalign{ df &= \vc{dF} \\ &= dx-h\LR{A\k I_m}\,\vc{\LR{3\LR{XA}^{\h 2}-\o}\odot\bR{dX\,A}} \\ &= dx-h\LR{A\k I_m}\,\c{\Diag{\vc{3\LR{XA}^{\h 2}-\o}}}\,\vc{dX\,A} \\ &= dx - h\LR{A\k I_m}\,\c{\D}\,\LR{A^T\k I_m}dx \\ J\doteq\grad fx &= I_{mr} - h\LR{A\k I_m}\,\c{\D}\,\LR{A^T\k I_m} \;\in\; \R{mr\times mr} }$$ NB: The Jacobian is a symmetric matrix.

Newton's method requires calculating $J^{-1}\,$ at every step, which is a standard operation for a matrix, but what does that mean for a fourth-order tensor?

IOW, you could calculate an expression for $\large\grad FX,\,$ but what would you do with it?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.