I am trying to solve a matrix equation in the following discrete form: $$ \frac{K^{n+1}-K^n}{\Delta t} = [(K^{n+1} (V^{n})^T).^3 - K^{n+1} (V^{n})^T]V^n. $$ where $K^{n+1} \in \mathbb{R}^{m \times r}, V \in \mathbb{R}^{n \times r}$, $V^T V = I$(identity matrix) and $.^3$ are elementwise operations.

I am attempting to solve using Newton iterate method, which means I need to differentiate the function F: $$ F = K^{n+1}-K^n-\Delta t[(K^{n+1} (V^{n})^T).^3 - K^{n+1} (V^{n})^T]V^n. $$ I differentiate F with respect to $K^{n+1}$ and get: $$ \frac{\partial F}{ \partial K} = I_K - \Delta t [3(K^{n+1} (V^{n})^T).^2 (V^{n})^T - (V^{n})^T]V^n. $$ In my understanding, if a matrix is differentiated with respect to itself, the result should be an identity matrix. However, since $K \in \mathbb{R}^{m \times r}$ is not a square matrix, how should this be handled? How can I solve the matrix equation? Thanks in advance.

  • 1
    $\begingroup$ The directional derivative of $A^3$ in direction $B$ is $A^2B+ABA+BA^2$, there is no further reduction as the matrix multiplication is not commutative. $\endgroup$ Mar 1 at 7:01
  • $\begingroup$ @LutzLehmann That was my first reaction, too, but then I noticed that those in OP's question are elementwise products, not matrix products. $\endgroup$ Mar 1 at 8:01
  • $\begingroup$ Ok, if that is indeed a compact form to write the discretized version of a PDE, then the factor 3 is correct. But then also, as per the answer, the identity matrix is not correct. $\endgroup$ Mar 1 at 8:35

2 Answers 2


If a matrix is differentiated with respect to itself, the result should be a fourth order tensor. The easist way to see this is to work with components.

$$ \frac{ \partial K_{ij}}{\partial {K_{kl}}} = \delta_{ik}\delta_{jl} $$

$\delta_{ij}$ is the Kronecker delta. Because differentiation survives only when $i=k$ and $j=k$. If your matrix is symmetric, then

$$ \frac{ \partial K_{ij}}{\partial {K_{kl}}} = \delta_{ik}\delta_{jl} + \delta_{il}\delta_{jk} $$

Because, the differentiation also survives for $i=l$ and $j=k$.


The correct answer for a symmetric matrix is

$$ \frac{ \partial K_{ij}}{\partial {K_{kl}}} = \frac{1}{2} (\delta_{ik}\delta_{jl} + \delta_{il}\delta_{jk} ) $$

See this

  • $\begingroup$ Thank you for your answer. If $K \in \mathbb{R}^{m \times n}$, then the tensor corresponding to $\partial K$ is of size $m \times n \times m \times n$, is that correct? $\endgroup$
    – Owen Jun
    Mar 1 at 5:04
  • $\begingroup$ Yes, I think that's correct, that's the standard ordering for $\frac{ \partial K_{ij}}{\partial {K_{kl}}}$ $\endgroup$
    – NNN
    Mar 1 at 5:41
  • 1
    $\begingroup$ If you are just interested in the directional derivative in a direction $H$, though, i.e., $\frac{d}{dt} F(K+tH)$ that's an $m\times n$ matrix (which is a function of $H$). $\endgroup$ Mar 1 at 11:37
  • $\begingroup$ I'm pretty sure I'm missing a factor of 1/2 in the symmetric case. $\endgroup$
    – NNN
    Mar 2 at 7:01
  • $\begingroup$ @FedericoPoloni I would be grateful for a better explanation of the symmetric matrix derivative wrt itself, than the one I linked to. $\endgroup$
    – NNN
    Mar 2 at 10:49

$ \def\R#1{{\mathbb R}^{#1}} \def\o{{\large\tt1}} \def\D{{\cal D}} \def\k{\otimes} \def\h{\odot} \def\bR#1{\big(#1\big)} \def\BR#1{\Big(#1\Big)} \def\LR#1{\left(#1\right)} \def\op#1{\operatorname{#1}} \def\vc#1{\op{vec}\LR{#1}} \def\Diag#1{\op{Diag}\LR{#1}} \def\qiq{\quad\implies\quad} \def\mt{\mapsto} \def\p{\partial} \def\grad#1#2{\frac{\p #1}{\p #2}} \def\c#1{\color{red}{#1}} \def\CLR#1{\c{\LR{#1}}} $First let's clean up some variable names by stripping off unnecessary decorations $$\eqalign{ \LR{\Delta t,V^T,K^{n+1},K^n} \;\mt\; \LR{h,A,X,Y} \\ }$$ Let's also use standard notation for elementwise products and powers $$\eqalign{ X^{\h 3} = X\h X\h X \\ }$$ And introduce the all-ones matrix $\o\in\R{m\times n}$ and identity matrix $I_m\in\R{m\times m}$

Rewrite the function and calculate its differential $$\eqalign{ F &= X - Y - hI_m\BR{\LR{XA}^{\h 3} - XA}A^T \\ dF &= dX - hI_m\BR{\LR{3\LR{XA}^{\h 2} - \o}\odot\bR{dX\,A}}A^T \\ }$$ To avoid introducing tensors, vectorize the equation and calculate the Jacobian $$\eqalign{ df &= \vc{dF} \\ &= dx-h\LR{A\k I_m}\,\vc{\LR{3\LR{XA}^{\h 2}-\o}\odot\bR{dX\,A}} \\ &= dx-h\LR{A\k I_m}\,\c{\Diag{\vc{3\LR{XA}^{\h 2}-\o}}}\,\vc{dX\,A} \\ &= dx - h\LR{A\k I_m}\,\c{\D}\,\LR{A^T\k I_m}dx \\ J\doteq\grad fx &= I_{mr} - h\LR{A\k I_m}\,\c{\D}\,\LR{A^T\k I_m} \;\in\; \R{mr\times mr} }$$ NB: The Jacobian is a symmetric matrix.

Newton's method requires calculating $J^{-1}\,$ at every step, which is a standard operation for a matrix, but what does that mean for a fourth-order tensor?

IOW, you could calculate an expression for $\large\grad FX,\,$ but what would you do with it?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.