Is there a relationship between the covariance matrix and the partial derivative?

Question

Suppose that there are $N$ pieces of data, each of which contains $M$ parameter values such that $N >> M$. If we put this information into matrix form ($N$ rows, $M$ columns) and then compute the covariance matrix of this data, $C$, then we can consider that $C_{i,j}$ is the covariance between parameter $i$, $p_i$, and parameter $j$, $p_j$.

I have always thought of this as "The amount that $p_i$ would change if $p_j$ changed by 1". While this may not be exactly correct (is it correct?), there is at least a fundamental relationship between $p_i$ and $p_j$ which is reflected in $C_{i,j}$ and can consider that if $C_{i,j} = 0$, then there is no fundamental relationship and the change in one will not necessarily reflect a change in the other.

Now, consider $\frac{\partial{p_i}}{\partial{p_j}}$, this is the amount of change in $p_i$ due to a small change in $p_j$ and this partial derivative will once again be zero if there is no relationship between $p_i$ and $p_j$.

Is it possible to consider, in any way, $C_{i,j} \approx \frac{\partial{p_i}}{\partial{p_j}}$? Are there conditions under which this may be the case or any other relationship between these two measures?

Answer 1

Let $\mathbf{\theta}$ be a Gaussian random vector with mean $\mathbf{\mu}$ and covariance matrix $\mathbf{\Sigma}_\mathbf{\theta}$. Let $\mathbf{p}_\theta$ denote the joint PDF.

Let $J_\mathbf{\theta}$ be the objective function, as its negative logarithm:

$$J_\mathbf{\theta}=-ln(\mathbf{p}_\theta)$$

By taking the partial derivatives w.r.t. $\theta_d$ and $\theta_{d'}$, the $(d,d')$ component of the Hessian can be obtained:

$$H^{(d,d')}(\mathbf{\mu})=\frac{\partial{J(\theta)}}{\partial{\theta_{d}}\partial{\theta_{d'}}}\Bigr|_{\theta=\mu}=(\mathbf\Sigma^{-1}_{\theta})^{(d,d')}$$

In that case, the Hessian matrix is equal to the inverse of the covariance matrix:

$$H(\mu)=\mathbf\Sigma^{-1}_{\theta}$$

Note that this is the case when evaluated at the mean value. Yet, for Gaussian random variables, the second derivatives of $J_\mathbf{\theta}$ are constant for all $\theta$ because of the quadratic nature. Therefore, Hessian matrix can be computed without obtaining the mean vector $\mathbf{\mu}$. It is also the case that the entries in Hessian matrix carry valuable geometric information about the random vector, such as curvature and conditional variances and etc.