Midterm Review
Math 123: Mathematical Aspects of Data Analysis – Fall 2026
I want to give a very quick overview of some of the threads between everything we’ve discussed. In particular, since the SVD arrived, every lecture has used it: best low-rank approximation, PCA, least squares, the statistics of least squares, ridge regression, and \(k\)-means (through spectral clustering but also more). Today we step back and see that almost every argument is something like
- Rotate into singular coordinates. In the bases \(\bfv_1,\bfv_2,\ldots\) and \(\bfu_1,\bfu_2,\ldots\), things dramatically simplify;
- Expand a square and kill the cross term. This turns a single error into “total = kept + lost”.
The three forms of the SVD
Throughout, \(\bfA\in\Rbb^{m\times n}\) has rank \(r\) and singular values \(\sigma_1\geq\cdots\geq\sigma_r>0\).
| Form | Formula |
|---|---|
| Full | \(\bfA=\bfU\Sigma\bfV^\top\), with \(\bfU\in\Rbb^{m\times m}\), \(\bfV\in\Rbb^{n\times n}\) orthogonal |
| Truncated | \(\bfA_k=\bfU_k\Sigma_k\bfV_k^\top\), \(k\leq r\) (and \(\bfA_r=\bfA\) is the compact SVD) |
| Rank-one sum | \(\bfA=\sum_{i=1}^r\sigma_i\bfu_i\bfv_i^\top\) |
I like to remember them through the rank-one sum form. Also, note the singular value “problem”, \[ \bfA\bfv_i=\sigma_i\bfu_i, \qquad \forall i = 1, \dots, r \]
Singular coordinates
One of the most reused fact in the course is \(\bfA\) in singular-vector coordinates. That is, expanding \(\bfx\) in the basis spanned by the right singular vectors, i.e., \(\bfx=\sum_{i=1}^n c_i\bfv_i\) with \(c_i = \bfv_i^\top \bfx\), then \[ \bfA\bfx=\sum_{i=1}^r\sigma_ic_i\bfu_i, \qquad \|\bfx\|_2^2=\sum_{i=1}^n c_i^2, \qquad \|\bfA\bfx\|_2^2=\sum_{i=1}^r\sigma_i^2c_i^2. \] This was consequential in understanding why
- \(\|\bfA\|_2 = \sigma_1\) and \(\|\bfA\|_F^2 = \sum_i \sigma_i^2\)
- Both Eckart-Young theorems
- The total variance decomposition
- Lots of the least squares + ridge regression theory
We can also use this to interpret least squares directly in the singular coordinates
Proof. Since \(\bfU\) and \(\bfV\) are orthogonal, \(\bfy=\bfU\bfb\) and \(\bftheta=\bfV\bfalpha\), and multiplying by \(\bfU^\top\) does not change lengths: \[ \|\bfy-\bfX\bftheta\|_2^2 =\|\bfU^\top(\bfy-\bfU\Sigma\bfV^\top\bftheta)\|_2^2 =\|\bfb-\Sigma\bfalpha\|_2^2. \] The vector \(\Sigma\bfalpha\in\Rbb^n\) has entries \(\sigma_i\alpha_i\) for \(i\leq r\) and \(0\) afterwards, which gives the first formula. The second is \(\|\bfV\bfalpha\|_2=\|\bfalpha\|_2\).
It is worth staring at this, since it reveals a lot of structure:
- The coordinates \(\alpha_{r+1},\ldots,\alpha_d\) do not appear in the error at all: they are the directions of \(\Ncal(\bfX)\), which the data cannot see.
- The coordinates \(b_{r+1},\ldots,b_n\) do not depend on \(\bftheta\) at all: they are the part of \(\bfy\) outside \(\Ccal(\bfX)\), which no fit can reach.
- What is left is \(r\) separate scalar problems, the \(i\)-th involving only \(\alpha_i\).
Least squares and ridge
Each of those scalar problems is a one-variable quadratic:
- Least squares: \((b_i-\sigma_i\alpha_i)^2\) is minimized by \(\alpha_i=b_i/\sigma_i\).
- Ridge: \((b_i-\sigma_i\alpha_i)^2+\lambda\alpha_i^2\) is minimized by \(\alpha_i=\sigma_ib_i/(\sigma_i^2+\lambda)\).
For \(i>r\), the minimum-norm solution and ridge both set \(\alpha_i=0\). Rotating back with \(\bftheta=\sum_i\alpha_i\bfv_i\) gives the formulas from lecture, \[ \bftheta^\dagger=\sum_{i=1}^r\frac{\bfu_i^\top\bfy}{\sigma_i}\,\bfv_i, \qquad \widehat{\bftheta}_\lambda=\sum_{i=1}^r\underbrace{\frac{\sigma_i^2}{\sigma_i^2+\lambda}}_{f_i}\cdot\frac{\bfu_i^\top\bfy}{\sigma_i}\,\bfv_i. \] So ridge is least squares with each term scaled by a filter factor \(f_i\in(0,1)\), which is near \(1\) when \(\sigma_i^2\gg\lambda\) and near \(0\) when \(\sigma_i^2\ll\lambda\).
A small \(\sigma_i\) is a direction \(\bfv_i\) along which the data barely vary. Least squares divides by \(\sigma_i\) there, so the variance \(\sigma^2/\sigma_i^2\) blows up. Ridge caps the gain, \(\sigma_i/(\sigma_i^2+\lambda)\leq 1/(2\sqrt{\lambda})\) however small \(\sigma_i\) is, at the price of some bias.
The cross term always vanishes
For any vectors, \(\|\bfa+\bfb\|_2^2=\|\bfa\|_2^2+2\bfa^\top\bfb+\|\bfb\|_2^2\), and an astonishing number of results in this course are this identity plus a reason why \(\bfa^\top\bfb=0\) (or has expectation zero).
| Result | Split | Why the cross term is zero |
|---|---|---|
| Orthogonal projection | \(\|\bfx\|_2^2=\|\bfP_S\bfx\|_2^2+\|\bfx-\bfP_S\bfx\|_2^2\) | residual \(\perp S\) |
| PCA | \(\operatorname{TV}=\operatorname{TV}(S)+\frac1n\Ecal(S)\) | the projection identity, row by row |
| Least squares | \(\|\bfy\|_2^2=\|\widehat{\bfy}\|_2^2+\|\widehat{\bfr}\|_2^2\) | \(\widehat{\bfr}\perp\Ccal(\bfX)\) |
| Minimum-norm solution | \(\|\bftheta^\dagger+\bfz\|_2^2=\|\bftheta^\dagger\|_2^2+\|\bfz\|_2^2\) | \(\bftheta^\dagger\in\Ncal(\bfX)^\perp\), \(\bfz\in\Ncal(\bfX)\) |
| Bias-variance | \(\operatorname{MSE}=\|\operatorname{Bias}\|_2^2+\operatorname{tr}\Cov\) | \(\Ebb[\widehat{\bftheta}-\Ebb\widehat{\bftheta}]=\mathbf{0}\) |
| Approximate Carathéodory | \(\Ebb\|\hat{\bfx}-\bfx\|_2^2=\frac1{k^2}\sum_j\Ebb\|\bfX_j-\bfx\|_2^2\) | independence and \(\Ebb[\bfX_j-\bfx]=\mathbf{0}\) |
| Sum of squares around a point | \(\sum_{i\in S}\|\bfx_i-\bfmu\|_2^2=\sum_{i\in S}\|\bfx_i-\bar{\bfx}_S\|_2^2+|S|\|\bar{\bfx}_S-\bfmu\|_2^2\) | \(\sum_{i\in S}(\bfx_i-\bar{\bfx}_S)=\mathbf{0}\) |
There are really only two reasons: orthogonality (geometry), and mean zero (statistics, or averaging over a set). The last row is the bias-variance decomposition with an average over \(S\) in place of an expectation, and for \(d=1\) it is fitting a constant by least squares.
k-means through the same lens
\(k\)-means is a sum-of-squares problem, so both ideas above apply.
Total = within + between
Summing the sum of squares around a point over the clusters, with \(\bfmu=\bar{\bfx}\), gives total = within + between: \[ \underbrace{\sum_{i=1}^n\|\bfx_i-\bar{\bfx}\|_2^2}_{n\cdot\operatorname{TV}} =\underbrace{\sum_{\ell=1}^k\sum_{i\in S_\ell}\|\bfx_i-\bar{\bfx}_{S_\ell}\|_2^2}_{\operatorname{WCSS}} +\underbrace{\sum_{\ell=1}^k|S_\ell|\,\|\bar{\bfx}_{S_\ell}-\bar{\bfx}\|_2^2}_{\operatorname{BCSS}}. \] This is the same shape as PCA’s “total = retained + lost”: the left side does not depend on the choice, so making one piece small is the same as making the other large. Tight clusters and well-separated clusters are one goal.
k-means as a matrix factorization
Encode a \(k\)-partition \(S_1,\ldots,S_k\) by its membership matrix \(\bfH\in\{0,1\}^{n\times k}\), with \(H_{i\ell}=1\) if \(i\in S_\ell\) and \(0\) otherwise, so each row of \(\bfH\) is a standard basis vector \(\bfe_\ell^\top\). Stack the centers as the rows of \(\bfM\in\Rbb^{k\times d}\). Then row \(i\) of \(\bfH\bfM\) is \(\bfmu_\ell^\top\) for the cluster \(\ell\) containing \(i\), so \[ \operatorname{cost}(S_1,\ldots,S_k;\bfmu_1,\ldots,\bfmu_k)=\sum_{\ell=1}^k\sum_{i\in S_\ell}\|\bfx_i-\bfmu_\ell\|_2^2=\|\bfX-\bfH\bfM\|_F^2. \] \(k\)-means approximates the data matrix by a product \(\bfH\bfM\) in which \(\bfH\) is constrained to be a membership matrix.
Fixing \(\bfH\) turns this into an ordinary least squares problem in \(\bfM\), one column (feature) at a time. The columns of \(\bfH\) are the indicator vectors of the clusters, so they are orthogonal and \(\bfH^\top\bfH=\operatorname{diag}(|S_1|,\ldots,|S_k|)\) is invertible. The normal equations give \[ \bfM=(\bfH^\top\bfH)^{-1}\bfH^\top\bfX, \] whose \(\ell\)-th row is \(\frac1{|S_\ell|}\sum_{i\in S_\ell}\bfx_i^\top=\bar{\bfx}_{S_\ell}^\top\). So the update step of Lloyd’s algorithm is a regression of each feature on the cluster indicators, that is, fitting one constant per cluster. The assignment step then minimizes over \(\bfH\) with \(\bfM\) fixed, so Lloyd’s algorithm is alternating minimization, and neither step can increase the cost.
The PCA lower bound
Without the constraint on \(\bfH\), the best approximation of \(\bfX\) by a product with \(k\) columns in the middle is the truncated SVD, by Eckart–Young. Done carefully, after centering, this compares \(k\)-means with PCA.
| Approximation of \(\bfX_c\) | Constraint | Rank | Best error | |
|---|---|---|---|---|
| PCA, \(k-1\) components | \(\bfZ\bfV_{k-1}^\top\) | none | \(\leq k-1\) | \(\sum_{j\geq k}\sigma_j^2\) (Eckart–Young) |
| \(k\)-means, \(k\) clusters | \(\bfH\bfB\), rows of \(\bfB\) are \((\bfmu_\ell-\bar{\bfx})^\top\) | \(\bfH\) a membership matrix | \(\leq k-1\) | \(\operatorname{opt}_k\) |
\(k\)-means is a constrained competitor in the Eckart–Young problem, so it cannot beat the unconstrained winner.
Proof. Fix a \(k\)-partition with means \(\bfmu_\ell=\bar{\bfx}_{S_\ell}\), membership matrix \(\bfH\), and center matrix \(\bfM\). Since every row of \(\bfH\) sums to one, \(\bfH\mathbf{1}_k=\mathbf{1}_n\), and so \[ \bfX-\bfH\bfM=(\bfX-\mathbf{1}_n\bar{\bfx}^\top)-(\bfH\bfM-\bfH\mathbf{1}_k\bar{\bfx}^\top)=\bfX_c-\bfH\bfB, \qquad \bfB=\bfM-\mathbf{1}_k\bar{\bfx}^\top. \] The rows of \(\bfB\) are \((\bfmu_\ell-\bar{\bfx})^\top\), and they satisfy one linear relation with non-zero coefficients: \[ \sum_{\ell=1}^k|S_\ell|(\bfmu_\ell-\bar{\bfx})=\sum_{i=1}^n\bfx_i-n\bar{\bfx}=\mathbf{0}. \] So the \(k\) rows of \(\bfB\) are linearly dependent, \(\operatorname{rank}(\bfB)\leq k-1\), and therefore \(\operatorname{rank}(\bfH\bfB)\leq k-1\). By Eckart–Young in the Frobenius norm applied to \(\bfX_c\), \[ \operatorname{WCSS}(S_1,\ldots,S_k)=\|\bfX_c-\bfH\bfB\|_F^2\geq\min_{\operatorname{rank}(\bfC)\leq k-1}\|\bfX_c-\bfC\|_F^2=\sum_{j\geq k}\sigma_j^2. \] Minimize over partitions and use optimal centers and optimal assignments. For \(k=1\) there is only one partition, with \(\operatorname{opt}_1=\|\bfX_c\|_F^2=\sum_j\sigma_j^2\).
In words: the \(k\) cluster means, recentered, span at most a \((k-1)\)-dimensional subspace, so \(k\)-means can never beat PCA with \(k-1\) components. Watch the index: the sum starts at \(\sigma_k\), not \(\sigma_{k+1}\), because \(k\) points span only a \((k-1)\)-dimensional affine subspace. This is one reason why projecting the data onto its top few principal directions before running \(k\)-means is a common preprocessing step: it makes each iteration cheaper, and the directions it discards are ones along which the clusters cannot be very spread out anyway.
Remark (Relaxation). Dropping the constraint “\(\bfH\) is a membership matrix” is the same relax and round strategy we used for spectral clustering, now applied to a point cloud instead of a graph. The relaxed problem is PCA, solved by singular vectors, and Theorem 1 is the lower bound that every relaxation provides.
Summary
| Problem | Objective | SVD answer |
|---|---|---|
| Low-rank approximation | \(\min_{\operatorname{rank}\bfB\leq k}\|\bfA-\bfB\|_F^2\) | \(\bfA_k=\sum_{i\leq k}\sigma_i\bfu_i\bfv_i^\top\), error \(\sum_{i>k}\sigma_i^2\) |
| PCA | \(\min_{\dim S\leq k}\Ecal(S)\) | \(S=\operatorname{span}\{\bfv_1,\ldots,\bfv_k\}\) of \(\bfX_c\), error \(\sum_{j>k}\sigma_j^2\) |
| Least squares | \(\min_{\bftheta}\|\bfy-\bfX\bftheta\|_2^2\) | \(\alpha_i=b_i/\sigma_i\), variance \(\sigma^2/\sigma_i^2\) |
| Ridge | \(\min_{\bftheta}\|\bfy-\bfX\bftheta\|_2^2+\lambda\|\bftheta\|_2^2\) | \(\alpha_i=f_i\,b_i/\sigma_i\), \(f_i=\sigma_i^2/(\sigma_i^2+\lambda)\) |
| \(k\)-means | \(\min_{\bfH,\bfM}\|\bfX-\bfH\bfM\|_F^2\) | no closed form, but \(\operatorname{opt}_k\geq\sum_{j\geq k}\sigma_j^2\) |
A checklist for when you are stuck:
- Write everything in singular coordinates, \(\bfx=\sum_ic_i\bfv_i\) and \(\bfy=\sum_ib_i\bfu_i\), and see whether the problem splits into scalar problems.
- If there is a square of a sum, expand it and find the reason the cross term is zero: orthogonality, or mean zero.
- Use orthonormality, \(\bfu_i^\top\bfu_j=\bfv_i^\top\bfv_j=\delta_{ij}\), and the cyclic property of the trace to clean up what is left.