This might be a silly quesntion but recently I've been trying to program the eigenface algorithm using PCA, so I arranged the face vectors vertically in a matrix X such as: X = [x1,x2,x3,...,xn]; In this case, what would be the wright way of computing the covariance matrix? cov(X) or cov(X')?

  • $\begingroup$ Depends on how the covariance function is implemented in your specific programming environment, but with data along the columns you should have $\Sigma = \frac{1}{n-1}(X-\mu\mathbf{1}^T)(X-\mu\mathbf{1}^T)^T$ if you want to do it yourself $\endgroup$
    – whpowell96
    Oct 3 '19 at 2:18
  • $\begingroup$ Thanks for the comment, but one thing I have trouble understanding is why isn't the covariance in this case the inner product of X with itself, is there any intuitive explanation? $\endgroup$
    – Marcus
    Oct 3 '19 at 2:53
  • 2
    $\begingroup$ Note that to compute the basis you should not form the covariance matrix, but instead compute the SVD of X (with or without removing its average first, it doesn’t matter much in practice). Then the columns of U are your basis. Computing the product X^T X directly is bound to introduce round-off errors, which tend to corrupt the basis vectors, especially the ones associated with the smaller singular values. $\endgroup$ Oct 3 '19 at 3:51
  • $\begingroup$ Do you have any sugestion of books I could read on the issue? $\endgroup$
    – Marcus
    Oct 3 '19 at 11:15
  • 1
    $\begingroup$ Also for a more heuristic argument, if we had a matrix of inner products, the resulting matrix would be size $n\times n$, but $n$ has nothing to do with the underlying random process, it is just the number of realizations we have access too. Any covariance matrix would have to be size $m\times m$ where $m$ is the dimension of each data point $x_i$. $\endgroup$
    – whpowell96
    Oct 3 '19 at 19:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.