Projections and the Principal Component Analysis

Math 123: Mathematical Aspects of Data Analysis – Fall 2026

Recap

Last time we proved the Eckart-Young theorem in both the spectral and Frobenius norms, then set up the geometric problem: for centered data, find the subspace of dimension at most \(k\) that best fits the point cloud. We stopped right after recalling orthogonal projection, so we restate the definition and its properties here before picking the thread back up.

Definition 1 (Orthogonal projection) Let \(S\subseteq\Rbb^d\) be a subspace with orthonormal basis \(\bfw_1,\ldots,\bfw_k\), and write \(\bfW=[\bfw_1\ \cdots\ \bfw_k]\in\Rbb^{d\times k}\), so that \(\bfW^\top\bfW=\bfI_k\). The orthogonal projection of \(\bfx\in\Rbb^d\) onto \(S\) is \[ \bfP_S\bfx =\sum_{i=1}^k(\bfw_i^\top\bfx)\bfw_i, \qquad \bfP_S=\bfW\bfW^\top\in\Rbb^{d\times d}. \] It keeps the component of \(\bfx\) in \(S\) and removes the component perpendicular to \(S\).

Proposition 1 (Properties of orthogonal projection) Let \(S\subseteq\Rbb^d\) be a subspace of dimension \(k\), with \(\bfP_S=\bfW\bfW^\top\) as in Definition 1, and let \(\bfx\in\Rbb^d\). Then:

1. \(\bfP_S^\top=\bfP_S\) and \(\bfP_S^2=\bfP_S\): projecting twice does the same thing as projecting once.
2. \(\operatorname{rank}(\bfP_S)=k\), and \(\bfP_S\) does not depend on which orthonormal basis of \(S\) is used.
3. \(\bfP_S\bfx\) is the unique closest point of \(S\) to \(\bfx\), that is, \(\|\bfx-\bfP_S\bfx\|_2<\|\bfx-\bfs\|_2\) for every \(\bfs\in S\) with \(\bfs\neq\bfP_S\bfx\).
4. The residual \(\bfx-\bfP_S\bfx\) is perpendicular to \(S\), so by the Pythagorean theorem, \[ \|\bfx\|_2^2=\|\bfP_S\bfx\|_2^2+\|\bfx-\bfP_S\bfx\|_2^2. \]
5. \(\bfI_d-\bfP_S\) is the orthogonal projection onto \(S^\perp\).

We also carry over the standing notation from last time: \(\bfX\in\Rbb^{n\times d}\) is the centered data matrix, \(r=\operatorname{rank}(\bfX)\leq\min\{n,d\}\), and \[ \bfX=\bfU_r\Sigma_r\bfV_r^\top=\sum_{j=1}^r\sigma_j\bfu_j\bfv_j^\top, \qquad \sigma_1\geq\cdots\geq\sigma_r>0, \] with \(\bfu_j\in\Rbb^n\) and \(\bfv_j\in\Rbb^d\), \(\bfV_k=[\bfv_1\ \cdots\ \bfv_k]\in\Rbb^{d\times k}\), \(\bfU_k=[\bfu_1\ \cdots\ \bfu_k]\in\Rbb^{n\times k}\), and \(\bfX_k=\sum_{j=1}^k\sigma_j\bfu_j\bfv_j^\top\) for the truncated SVD of \(\bfX\).

The best-fitting subspace

Among all subspaces of dimension at most \(k\), which one passes closest to the point cloud, measuring “close” by total squared distance?

Definition 2 (Reconstruction error) Let \(S\subseteq\Rbb^d\) be a subspace. The reconstruction error of the centered data on \(S\) is \[ \Ecal(S) =\sum_{i=1}^n\|(\bfx_i-\bar{\bfx})-\bfP_S(\bfx_i-\bar{\bfx})\|_2^2. \] A best-fitting \(k\)-dimensional subspace is a minimizer of \(\Ecal(S)\) over subspaces \(S\subseteq\Rbb^d\) with \(\dim S\leq k\).

Each term is the squared distance from the centered observation to the subspace (Proposition 1, part 3), so \(\Ecal(S)\) is exactly the total squared distance from the data to \(S\).

We can write the above definition in the following (familiar!) matrix form.

Proposition 2 (Reconstruction error is a Frobenius norm) Let \(S\subseteq\Rbb^d\) have orthonormal basis matrix \(\bfW\in\Rbb^{d\times k}\), so \(\bfP_S=\bfW\bfW^\top\). Then \(\bfX\bfP_S\) is the matrix whose \(i\)-th row is \((\bfP_S(\bfx_i-\bar{\bfx}))^\top\), it has rank at most \(k\), and \[ \Ecal(S) =\|\bfX-\bfX\bfP_S\|_F^2 =\|\bfX-\bfX\bfW\bfW^\top\|_F^2. \]

NoteProof

Proof. Row \(i\) of \(\bfX\bfP_S\) is \((\bfx_i-\bar{\bfx})^\top\bfP_S=(\bfP_S(\bfx_i-\bar{\bfx}))^\top\), using \(\bfP_S^\top=\bfP_S\). The squared Frobenius norm of a matrix is the sum of the squared norms of its rows (the Frobenius norm sums the squared entries, which we group by row), so \(\|\bfX-\bfX\bfP_S\|_F^2\) is exactly the sum defining \(\Ecal(S)\). Finally \(\bfX\bfP_S=(\bfX\bfW)\bfW^\top\) is a product with an inner dimension of \(k\), so its rank is at most \(k\).

The geometric problem is now a rank-constrained matrix approximation problem, and we know the answer to those! Enter Eckart-Young:

Theorem 1 (Best-fitting subspaces come from the SVD) Let \(1\leq k\leq r\) and set \[ S_k=\operatorname{span}\{\bfv_1,\ldots,\bfv_k\}, \qquad \bfP_{S_k}=\bfV_k\bfV_k^\top. \] Then \(S_k\) is a best-fitting \(k\)-dimensional subspace, projection onto it reproduces the truncated SVD, \[ \bfX\bfV_k\bfV_k^\top=\bfX_k, \] and the optimal error is \[ \min_{\dim S\leq k}\Ecal(S) =\Ecal(S_k) =\sum_{j=k+1}^r\sigma_j^2. \]

NoteProof

Proof. For the identity, use \(\bfX\bfv_j=\sigma_j\bfu_j\) (building the left singular vectors): \[ \bfX\bfV_k\bfV_k^\top =\sum_{j=1}^k(\bfX\bfv_j)\bfv_j^\top =\sum_{j=1}^k\sigma_j\bfu_j\bfv_j^\top =\bfX_k. \] For the lower bound, let \(S\) be any subspace with \(\dim S\leq k\). By Proposition 2, \(\bfX\bfP_S\) has rank at most \(\dim S\leq k\), so \(\bfX\bfP_S\) is a competitor in the minimization of Eckart-Young theorem, Frobenius norm and \[ \Ecal(S)=\|\bfX-\bfX\bfP_S\|_F^2\geq\|\bfX-\bfX_k\|_F^2=\sum_{j=k+1}^r\sigma_j^2. \] The choice \(S=S_k\) turns this inequality into an equality, by the identity just proved.

Note, Eckart-Young minimizes over all rank-\(k\) matrices, a larger set than the projections \(\bfX\bfP_S\), and the winner happens to be a projection. That is what lets a theorem about matrix approximation answer a question about geometry.

Principal component analysis

Definition 3 (Principal component analysis) Let \(\bfX\) be centered with SVD as above and fix \(1\leq k\leq r\). Principal component analysis (PCA) with \(k\) components consists of:

• the principal directions (or loading vectors) \(\bfv_1,\ldots,\bfv_k\), the top \(k\) right singular vectors of \(\bfX\);
• the principal subspace \(S_k=\operatorname{span}\{\bfv_1,\ldots,\bfv_k\}\);
• the scores, the coordinates of each observation in that subspace, \[ \bfz_i=\bfV_k^\top(\bfx_i-\bar{\bfx})\in\Rbb^k, \qquad \bfZ=\bfX\bfV_k=\bfU_k\Sigma_k\in\Rbb^{n\times k}; \]
• the reconstructions \(\widehat{\bfx}_i=\bar{\bfx}+\bfV_k\bfz_i\), whose centered form is the matrix \(\bfX\bfV_k\bfV_k^\top=\bfX_k\).

The \(j\)-th principal component is the score vector \(\bfX\bfv_j=\sigma_j\bfu_j\in\Rbb^n\), the list of the \(j\)-th coordinates across the whole dataset.

By Theorem 1, PCA is not just a \(k\)-dimensional summary of the data; it is a best one in squared reconstruction error, and it is computed by a single SVD.

Warning“Principal component” is overloaded

Some authors call \(\bfv_j\) the \(j\)-th principal component, others reserve that name for the scores \(\bfX\bfv_j\), and software splits the difference (sklearn’s components_ are the \(\bfv_j\), while fit_transform returns the scores). We will say principal direction for \(\bfv_j\in\Rbb^d\) and score for the coordinates, and you should check the convention whenever you read someone else’s statement. Note also that principal directions are only determined up to sign, and not determined at all when singular values tie.

The variance interpretation

Definition 4 (Total variance) The total variance of the data is \[ \operatorname{TV} =\frac1n\sum_{i=1}^n\|\bfx_i-\bar{\bfx}\|_2^2 =\frac1n\|\bfX\|_F^2 =\frac1n\sum_{j=1}^r\sigma_j^2, \] using the Frobenius norm and all the singular values. For a subspace \(S\subseteq\Rbb^d\), the variance retained by \(S\) is \[ \operatorname{TV}(S)=\frac1n\|\bfX\bfP_S\|_F^2 =\frac1n\sum_{i=1}^n\|\bfP_S(\bfx_i-\bar{\bfx})\|_2^2. \]

Proposition 3 (Minimum error equals maximum retained variance) For every subspace \(S\subseteq\Rbb^d\), \[ \operatorname{TV} =\operatorname{TV}(S)+\frac1n\Ecal(S). \] Since \(\operatorname{TV}\) does not depend on \(S\), a subspace minimizes reconstruction error over \(\dim S\leq k\) if and only if it maximizes retained variance. In particular, the principal subspace \(S_k\) maximizes retained variance, with \[ \operatorname{TV}(S_k)=\frac1n\sum_{j=1}^k\sigma_j^2. \]

NoteProof

Proof. Apply Proposition 1, part 4, to each centered observation: \[ \|\bfx_i-\bar{\bfx}\|_2^2 =\|\bfP_S(\bfx_i-\bar{\bfx})\|_2^2+\|(\bfx_i-\bar{\bfx})-\bfP_S(\bfx_i-\bar{\bfx})\|_2^2. \] Sum over \(i\) and divide by \(n\). The last claim is Theorem 1 plus \(\|\bfX_k\|_F^2=\sum_{j\leq k}\sigma_j^2\).

So the principal directions can be described in two equivalent ways: minimize the distance you throw away, or keep as much spread as you can. Proposition 3 says these are equivalent.

Definition 5 (Explained variance ratio) Assume \(\bfX\neq\mathbf{0}\). The explained variance ratio of the first \(k\) components is \[ R_k=\frac{\operatorname{TV}(S_k)}{\operatorname{TV}} =\frac{\sum_{j=1}^k\sigma_j^2}{\sum_{j=1}^r\sigma_j^2}, \] and the relative squared reconstruction error is \(1-R_k\). A common rule of thumb is to take the smallest \(k\) with \(R_k\geq0.95\).

NoteWhat to plot to visualize your data in 2D or 3D

Take \(k=2\) (or \(k=3\)) in Definition 3, so the projection is onto the principal subspace \(S_2=\operatorname{span}\{\bfv_1,\bfv_2\}\) (or \(S_3=\operatorname{span}\{\bfv_1,\bfv_2,\bfv_3\}\)). By Theorem 1 this is the plane (or 3D subspace) that loses the least squared distance, which by Proposition 3 is the one that keeps the most spread.

The object you plot is the score matrix \[ \bfZ=\bfX\bfV_k=\bfU_k\Sigma_k\in\Rbb^{n\times k}, \qquad k\in\{2,3\}, \] so row \(i\) is the point \(\bfz_i=\bfV_k^\top(\bfx_i-\bar{\bfx})\in\Rbb^2\) (or \(\Rbb^3\)), one dot per observation, with horizontal coordinate \(\bfv_1^\top(\bfx_i-\bar{\bfx})\), vertical coordinate \(\bfv_2^\top(\bfx_i-\bar{\bfx})\), and, in 3D, depth \(\bfv_3^\top(\bfx_i-\bar{\bfx})\).

Be careful not to confuse this with the projected data \(\bfX\bfV_k\bfV_k^\top=\bfX_k\in\Rbb^{n\times d}\). Its rows are the same projected points, but written in the original \(d\) coordinates, so they still live in \(\Rbb^d\) and cannot be drawn directly. The \(k\times d\) matrix \(\bfV_k^\top\) is what maps each centered observation to plotting coordinates.

Note, nothing is distorted inside the subspace! Since \(\bfV_k\) has orthonormal columns, \(\|\bfV_k\bfz_i-\bfV_k\bfz_j\|_2=\|\bfz_i-\bfz_j\|_2\), so distances in the plot are exactly the distances between the projected points. What you do lose is everything perpendicular to \(S_k\), and the explained variance ratio \(R_2\) or \(R_3\) (Definition 5) tells you how much of the spread the picture shows. If it is small, points that look close in the plot may be far apart in \(\Rbb^d\).