4
$\begingroup$

Suppose that there are $N$ pieces of data, each of which contains $M$ parameter values such that $N >> M$. If we put this information into matrix form ($N$ rows, $M$ columns) and then compute the covariance matrix of this data, $C$, then we can consider that $C_{i,j}$ is the covariance between parameter $i$, $p_i$, and parameter $j$, $p_j$.

I have always thought of this as "The amount that $p_i$ would change if $p_j$ changed by 1". While this may not be exactly correct (is it correct?), there is at least a fundamental relationship between $p_i$ and $p_j$ which is reflected in $C_{i,j}$ and can consider that if $C_{i,j} = 0$, then there is no fundamental relationship and the change in one will not necessarily reflect a change in the other.

Now, consider $\frac{\partial{p_i}}{\partial{p_j}}$, this is the amount of change in $p_i$ due to a small change in $p_j$ and this partial derivative will once again be zero if there is no relationship between $p_i$ and $p_j$.

Is it possible to consider, in any way, $C_{i,j} \approx \frac{\partial{p_i}}{\partial{p_j}}$? Are there conditions under which this may be the case or any other relationship between these two measures?

$\endgroup$
4
$\begingroup$

Let $\mathbf{\theta}$ be a Gaussian random vector with mean $\mathbf{\mu}$ and covariance matrix $\mathbf{\Sigma}_\mathbf{\theta}$. Let $\mathbf{p}_\theta$ denote the joint PDF.

Let $J_\mathbf{\theta}$ be the objective function, as its negative logarithm:

$$J_\mathbf{\theta}=-ln(\mathbf{p}_\theta)$$

By taking the partial derivatives w.r.t. $\theta_d$ and $\theta_{d'}$, the $(d,d')$ component of the Hessian can be obtained:

$$H^{(d,d')}(\mathbf{\mu})=\frac{\partial{J(\theta)}}{\partial{\theta_{d}}\partial{\theta_{d'}}}\Bigr|_{\theta=\mu}=(\mathbf\Sigma^{-1}_{\theta})^{(d,d')}$$

In that case, the Hessian matrix is equal to the inverse of the covariance matrix:

$$H(\mu)=\mathbf\Sigma^{-1}_{\theta}$$

Note that this is the case when evaluated at the mean value. Yet, for Gaussian random variables, the second derivatives of $J_\mathbf{\theta}$ are constant for all $\theta$ because of the quadratic nature. Therefore, Hessian matrix can be computed without obtaining the mean vector $\mathbf{\mu}$. It is also the case that the entries in Hessian matrix carry valuable geometric information about the random vector, such as curvature and conditional variances and etc.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.