0
$\begingroup$

I want to perform gradient descent optimization of the probability of a sample under a multivariate normal probability density function. For your convenience I state the PDF here:

$$\mathcal{N}(\boldsymbol\mu,\,\boldsymbol\Sigma) = (2\pi)^{-\frac{k}{2}}|\boldsymbol\Sigma|^{-\frac{1}{2}}\, e^{ -\frac{1}{2}(\mathbf{x}-\boldsymbol\mu)'\boldsymbol\Sigma^{-1}(\mathbf{x}-\boldsymbol\mu) }$$

The probability of a sample under this model for the special case of zero mean and unit covariance is given by

$$p(\vec{x}) = \frac{1}{(2\pi)^{-n/2}}\prod_{i=1}^n e^{-\frac{x_i^2}{2}}$$

where $n$ is the number of entries in $\vec{x}$. $\vec{x}$ is the projection of a sample onto a PCA model in each iteration and so should actually be seen as a function of a different set of variables $\vec{y}$ that are parametrizing the original problem. In addition, I would like to work with the log of the probability for numerical stability. All in all, I'm looking for the derivative

$$\frac{d}{d\vec{y}}\log p(\vec{x(\vec{y}})) = \frac{d}{d\vec{y}}\log \frac{1}{(2\pi)^{-n/2}}\prod_{i=1}^n e^{-\frac{\vec{x_i(\vec{y}})^2}{2}}$$

and this where I'm stuck. Help and pointers are really appreciated. So my questions are these:

(1) Is this approach to deriving a gradient descent cost function correct?

(2) What is the derivative given by?

$\endgroup$
3
$\begingroup$

Start by simplifying your expression by using the fact that the log of a product is the sum of the logarithms of the factors in the product. The resulting expression is a quadratic form that is easy to differentiate.

$\endgroup$
  • $\begingroup$ Thanks! I guess the real source of my confusion was related to x being a function of y for which I don't have the analytical expression but I found a solution using the Jacobian. Your answer pointed me in the right direction so I'm accepting that as answer. $\endgroup$ – Snut Aug 14 '14 at 9:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.