# Derive PCA with SVD

The context is I have a big matrix, 20K * 50K, and I want reduce the dimensionality. In R, it's impossible to apply PCA with more variables(columns) than observations(rows). Therefore, I am trying a partial SVD with this matrix, for instance, I calculate the top 100 left and right singular vectors. But with these singular vectors, how can I obtain the new data. For example, for PCA, we have loadings for calculating the scores in the new transformed coordinate system, can we do the same thing with SVD?

Thanks for your kind response !

-

Suppose your data matrix is $\mathbf{X} \in \mathbb{R}^{n \times m}$, where $m$ is the number of data points, and $n$ is the dimensionality of the data. Let the SVD of $\mathbf{X}$ be

\begin{align} \mathbf{X} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}. \end{align}

Assume for the sake of simplicity that the data points that make up $\mathbf{X}$ have been not been translated (for example, if you normally subtract the mean, assume it's zero). Also, assume that the singular values are in decreasing order from left to right. Let $\mathbf{U}_{k}$ be the matrix consisting of the leftmost $k$ columns of $\mathbf{U}$ (i.e., the left singular vectors corresponding to the $k$ largest singular values). Then PCA defines the following transformations:

• Mapping into a lower dimensional coordinate system: given $\mathbf{x} \in \mathbb{R}^{n}$, where $n$ corresponds to the dimensionality of your original data, the mapped (or transformed) data will be $\mathbf{y} \in \mathbb{R}^{k}$, where $\mathbf{y} = \mathbf{U}_{k}^{T}\mathbf{x}$
• Mapping the lower dimensional, transformed data back to the original host space and coordinate system: given the transformed data point $\mathbf{y} \in \mathbb{R}^{k}$, the representation of that point in the original host space and coordinate system is $\tilde{\mathbf{x}} = \mathbf{U}_{k}\mathbf{y}$.
• Projecting onto a lower dimensional subspace: given $\mathbf{x} \in \mathbb{R}^{n}$, PCA defines an orthogonal projector $\mathbf{P}_{k} = \mathbf{U}_{k}\mathbf{U}_{k}^{T}$, so that the projected data point $\tilde{\mathbf{x}} \in \mathbb{R}^{n}$ is defined by $\tilde{\mathbf{x}} = \mathbf{P}_{k}\mathbf{x}$. Projection is equal to the second mapping in this list composed with the first mapping in this list.

If you translate all of your data by a point $\mathbf{x}_{0} \in \mathbb{R}^{n}$, just replace $\mathbf{x}$ in the above list with the expression $(\mathbf{x} - \mathbf{x}_{0})$, and the same expressions will hold.

-
 Thanks at first, but still I don't understand well your explanation. Especially the first point, i.e. mapping into a lower dimensional coordinate system. Suppose the X is a matrix 8 by 10, and we apply a partial SVD on it, taking only top 5 singular values, and singular right/left vectors. Then according to my experiment, U will be 8 by 5, and V is 10 by 5. Now I have a test dataset, A to be transformed, and its size is 2 by 10 (same columns as attributes), clearly the U^T won't have an appropriate size for matrix multiplication with A. Am I right ? – Ensom Hodder May 30 '12 at 14:56 I think you're treating data points as rows in your data matrix, but the explanation above assumes that each data point is a column of the data matrix, in which case you'd have to transpose matrices appropriately. – Geoff Oxberry♦ May 30 '12 at 15:00 You're right ! That's my fault :-p Anyway, thanks a lot for your kind response ! – Ensom Hodder May 30 '12 at 15:51 By the way, for someone uses row for observations, and columns for attributes(variables), we may obtain the transformed data by this formula : $\mathbf{y} = \mathbf{x}\mathbf{V}_{k}^{T}$ – Ensom Hodder May 30 '12 at 16:05

Checkout gensim, Latent Semantic Analysis is the field of linguistics code-word for iterative SVD.

-