2
$\begingroup$

In order to achieve a 2D representation $X\in\mathbb{R}^{n\times 2}$ of some high-dimensional data residing in $Y\in\mathbb{R}^{n\times k}$, I use PCA:$$X=Y\cdot U,$$where $U\in\mathbb{R}^{k\times 2}$ contains eigenvectors of $Y^TY$ corresponding to its dominant eigenvalues.

However, in case there are multiple occurrences of, e.g., first dominant eigenvalue, my PCA solution, as defined above, will be inconsistent: it would depend on the actual eigenvalue that is declared as 'the first dominant' by the method that I use for the eigendecomposition. What is the recipe to allow for a consistent solution?

Perhaps more important is the following. Namely, PCA provides guarantees on maximal variance along axes; what impact does the above problem have on the solution with maximal variance along axes? Will maximal variance be retained with each solution, regardless on which eigenvector corresponding to dominant first eigenvalue is used?

Improve this question
$\endgroup$
3
$\begingroup$

PCA is called this way since it picks the principal components. If you happen to have several components with the same or almost the same eigenvalue and you pick one but not the other, then you can't claim that you picked the principal components. You picked a subset of the principal components. In other words, if your second eigenvalue is doubled, then it's not a useful strategy to choose only two principal components, but you need to consider three.

Improve this answer
$\endgroup$
4
  • $\begingroup$ To summarize, the multiplicity of the last eigenvalue considered, $mult(\lambda)$, determines the number of principal components that should be additionally considered, which is $mult(\lambda)-1$. Is this correct? $\endgroup$ – usero Aug 6 '12 at 6:55
  • $\begingroup$ You phrase it in too complicated a way. The number of significant eigenvalues determines the number of components you need to consider. $\endgroup$ – Wolfgang Bangerth Aug 6 '12 at 7:44
  • $\begingroup$ Suppose a PCA solution corresponding to two dominant eigenvalues of the same magnitude is denoted by $X\in\mathbb{R}^{n\times 2}$. Then, besides axis swapping, any rotation of $X$ would also yield a 2D solution with the same variance across $x$ and $y$ axis, subject to axes orthogonality? $\endgroup$ – usero Nov 16 '12 at 15:14
  • $\begingroup$ Yes, that is correct. Because the eigenvalues are the same under rotation. $\endgroup$ – Wolfgang Bangerth Nov 17 '12 at 11:36
2
$\begingroup$

If the largest eigenvalue has multiplicity, the PCA solution really depends on your choice of the eigenvectors as 'the first two dominant'. Different choice will give you different representation of the original high-dimensional data, but they all will have the same maximal variance.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Please consider the comment below the accepted answer. Thanks. $\endgroup$ – usero Nov 16 '12 at 18:19
  • $\begingroup$ It's analogous to diagonalization of a matrix: if you choose different modal matrices, the diagonalized matrix will have diagonal elements with different orders, but those diagonalized matrices are similar, which means they are essential the same; it's just a choice of basis. $\endgroup$ – chaohuang Nov 16 '12 at 19:04
  • $\begingroup$ Ok, but that also implies different 1D solutions with maximal variance (so, not only the ones being eigenvectors corresponding to eigenvalues of same magnitude) $\endgroup$ – usero Nov 16 '12 at 19:16
  • 2
    $\begingroup$ I guess you should notice that eigenvectors corresponding to repeated eigenvalues are not unique; any linear combination of two eigenvectors is still the eigenvectors with respect to the same eigenvalue. $\endgroup$ – chaohuang Nov 16 '12 at 19:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.