How Low Can You Go: Eckart-Young

Math 123: Mathematical Aspects of Data Analysis – Fall 2026

Published

September 22, 2026

Recap

Last time we introduced two norms for measuring the size of a matrix, in order to make precise what it means for the truncated SVD \(\bfA_k\) to be a best rank-\(k\) approximation of \(\bfA\), and we stated the Eckart-Young theorem without proof. Throughout, \(\bfA\in\Rbb^{m\times n}\) has rank \(r\), compact SVD \(\bfA=\bfU_r\Sigma_r\bfV_r^\top=\sum_{i=1}^r\sigma_i\bfu_i\bfv_i^\top\) (SVD), and singular values \(\sigma_1\geq\cdots\geq\sigma_r>0\).

Recall the two norms: for \(\bfM\in\Rbb^{m\times n}\), the Frobenius norm is \[ \|\bfM\|_F =\big(\langle\bfM,\bfM\rangle_F\big)^{1/2} =\big(\operatorname{tr}(\bfM^\top\bfM)\big)^{1/2} =\left(\sum_{i=1}^m\sum_{j=1}^n M_{ij}^2\right)^{1/2}, \] and the spectral norm is \[ \|\bfM\|_2 =\max_{\|\bfx\|_2=1}\|\bfM\bfx\|_2. \]

The plan today is to fully crush questions of optimality. We’ll prove Eckart-Young in the spectral norm, then state the Frobenius-norm version and spend the rest of the lecture reading it geometrically: as the problem of finding the subspace that best fits a cloud of data points. That reading is principal component analysis.

Reading the spectral norm off the singular values

Lemma 1 (\(\bfA\) in singular-vector coordinates) Let \(\bfx\in\Rbb^n\), and write \(\bfx=\sum_{i=1}^n c_i\bfv_i\) with \(c_i=\bfv_i^\top\bfx\) (expansion in an orthonormal basis). Then \[ \bfA\bfx=\sum_{i=1}^r\sigma_ic_i\bfu_i, \qquad \|\bfx\|_2^2=\sum_{i=1}^n c_i^2, \qquad \|\bfA\bfx\|_2^2=\sum_{i=1}^r\sigma_i^2c_i^2. \]

Proof. Recall, \(\bfA\bfv_i=\sigma_i\bfu_i\) for \(i\leq r\) and \(\bfA\bfv_i=\mathbf{0}\) for \(i>r\) (building the left singular vectors). Therefore, \[ \bfA\bfx=\sum_{i=1}^n c_i\bfA\bfv_i=\sum_{i=1}^r\sigma_ic_i\bfu_i. \] For the two norms, expand the inner products and use orthonormality of \(\bfv_1,\ldots,\bfv_n\) and of \(\bfu_1,\ldots,\bfu_r\): \[ \|\bfx\|_2^2 =\sum_{i=1}^n\sum_{j=1}^n c_ic_j\,\bfv_j^\top\bfv_i =\sum_{i=1}^n c_i^2, \qquad \|\bfA\bfx\|_2^2 =\sum_{i=1}^r\sum_{j=1}^r\sigma_i\sigma_jc_ic_j\,\bfu_j^\top\bfu_i =\sum_{i=1}^r\sigma_i^2c_i^2, \] since the cross terms vanish and the diagonal terms have \(\bfv_i^\top\bfv_i=\bfu_i^\top\bfu_i=1\).

This insight into how we can describe the action of \(A\) through its SVD gives us a really clean perspective on the spectral norm.

Proposition 1 (The spectral norm is the largest singular value) \[ \|\bfA\|_2=\sigma_1. \]

Proof. Let \(\bfx\in\Rbb^n\) be a unit vector, with coordinates \(c_i\) as in Lemma 1. Since \(\sigma_i\leq\sigma_1\) for every \(i\leq r\), \[ \|\bfA\bfx\|_2^2 =\sum_{i=1}^r\sigma_i^2c_i^2 \leq\sigma_1^2\sum_{i=1}^r c_i^2 \leq\sigma_1^2\sum_{i=1}^n c_i^2 =\sigma_1^2\|\bfx\|_2^2 =\sigma_1^2. \] So \(\|\bfA\bfx\|_2\leq\sigma_1\) for every unit \(\bfx\). Equality holds at \(\bfx=\bfv_1\), since \(\bfA\bfv_1=\sigma_1\bfu_1\) and \(\|\bfu_1\|_2=1\). Hence the maximum over unit vectors is attained and equals \(\sigma_1\).

Eckart-Young in the spectral norm

We can now prove the first of the two theorems stated last time.

Theorem 1 (Eckart-Young theorem, spectral norm) If \(k\) is an integer satisfying \(0\leq k<r\), then the truncated SVD \(\bfA_k\) (rank-\(k\) truncated SVD) is a best rank-at-most-\(k\) approximation of \(\bfA\) in the spectral norm: \[ \min_{\operatorname{rank}(\bfB)\leq k} \|\bfA-\bfB\|_2 =\|\bfA-\bfA_k\|_2 =\sigma_{k+1}. \] When \(0<k<r\), a minimizing matrix need not be unique. For \(k\geq r\), the minimum is zero and is attained by \(\bfB=\bfA\).

Proof. Let \[ \bfA = \sum_{i=1}^r \sigma_i \bfu_i \bfv_i^\top, \] be a SVD of \(\bfA\) with its usual properties. We’ll also consider \(\{\bfv_1, \dots, \bfv_n\}\) as an orthonormal basis for \(\Rbb^n\) (recall, we get \(n\) instead of \(r\) by considering the spectral decomposition of \(\bfA^\top\bfA\)).

First, we show the second equality. Let \[ \bfR_k =\bfA-\bfA_k =\sum_{i=k+1}^r\sigma_i\bfu_i\bfv_i^\top. \] The vectors \(\bfu_{k+1},\ldots,\bfu_r\) and \(\bfv_{k+1},\ldots,\bfv_r\) are orthonormal so this expression for \(\bfR_k\) is itself a compact SVD, with largest singular value \(\sigma_{k+1}\). By Proposition 1 applied to \(\bfR_k\), \[ \|\bfA-\bfA_k\|_2=\|\bfR_k\|_2=\sigma_{k+1}. \]

Next, we show optimality of the truncated SVD. Let \(\bfB\in\Rbb^{m\times n}\) have rank at most \(k\). The main idea here is that, for any matrix of rank \(k\) like \(\bfB\), it must miss one of the top \(k+1\) singular vectors. Therefore, we can show that it incurs error of at least \(\sigma_{k+1}\). Since \(\bfA_k\) always attains exactly that error, it’s always as good as, if not better, than any other \(\bfB\) of rank \(k\).

To achieve the above vision, set \[ W=\operatorname{span}\{\bfv_1,\ldots,\bfv_{k+1}\}. \] To be formal about why \(\bfB\) must miss a vector in \(W\), consider rank-nullity. By rank-nullity, \(\dim\Ncal(\bfB)\geq n-k\), while \(\dim W=k+1\), so \[ \dim\big(\Ncal(\bfB)\cap W\big) \geq(n-k)+(k+1)-n=1, \] and \(\Ncal(\bfB)\cap W\) contains a nonzero vector. Choose a unit vector \(\bfw\) in this intersection (i.e., \(\|\bfw\| = 1)\).

By Lemma 1, its singular-vector coordinates must satisfy \(c_i=\bfv_i^\top\bfw=0\) for \(i>k+1\), because \(\bfw\in W\). Thus \(\sum_{i=1}^{k+1}c_i^2=\|\bfw\|_2^2=1\). Moreover, as \(\bfB\bfw=\mathbf{0}\) and \(\sigma_i\geq\sigma_{k+1}\) for \(i\leq k+1\), \[ \|\bfA-\bfB\|_2^2 \geq\|(\bfA-\bfB)\bfw\|_2^2 =\|\bfA\bfw\|_2^2 =\sum_{i=1}^{k+1}\sigma_i^2c_i^2 \geq\sigma_{k+1}^2\sum_{i=1}^{k+1}c_i^2 =\sigma_{k+1}^2. \] So \(\|\bfA-\bfB\|_2\geq\sigma_{k+1}\) for every rank-at-most-\(k\) matrix \(\bfB\), and this bound is attained by \(\bfB=\bfA_k\).

For \(k\geq r\), \(\bfA\) itself has rank at most \(k\), so the minimum is \(0=\|\bfA-\bfA\|_2\).

Eckart-Young in the Frobenius norm

The Frobenius norm was defined through the trace (Frobenius norm), so before we read \(\|\bfA\|_F\) off the singular values, let us collect the facts about the trace that we will use today and for the rest of the course.

Definition 1 (Trace) The trace of a square matrix \(\bfM\in\Rbb^{d\times d}\) is the sum of its diagonal entries, \[ \operatorname{tr}(\bfM)=\sum_{i=1}^d M_{ii}. \]

Like the determinant, the trace compresses a square matrix down to a single number. Unlike the determinant, it is cheap to compute and it interacts beautifully with sums and products.

Proposition 2 (Properties of the trace)  

  1. (Linearity) For \(\bfM,\bfN\in\Rbb^{d\times d}\) and \(\alpha,\beta\in\Rbb\), \[ \operatorname{tr}(\alpha\bfM+\beta\bfN) =\alpha\operatorname{tr}(\bfM)+\beta\operatorname{tr}(\bfN). \]
  2. (Transpose invariance) For \(\bfM\in\Rbb^{d\times d}\), \(\operatorname{tr}(\bfM^\top)=\operatorname{tr}(\bfM)\).
  3. (Cyclic property) For \(\bfB\in\Rbb^{m\times n}\) and \(\bfC\in\Rbb^{n\times m}\), both \(\bfB\bfC\) and \(\bfC\bfB\) are square, and \[ \operatorname{tr}(\bfB\bfC)=\operatorname{tr}(\bfC\bfB), \] even though \(\bfB\bfC\) and \(\bfC\bfB\) need not have the same size.
  4. (Rank-one matrices) For \(\bfx,\bfy\in\Rbb^d\), \[ \operatorname{tr}(\bfx\bfy^\top)=\bfy^\top\bfx, \qquad\text{in particular}\qquad \operatorname{tr}(\bfx\bfx^\top)=\|\bfx\|_2^2. \]

Proof. For part 1, the diagonal entry of \(\alpha\bfM+\beta\bfN\) in position \(i\) is \(\alpha M_{ii}+\beta N_{ii}\), so \[ \operatorname{tr}(\alpha\bfM+\beta\bfN) =\sum_{i=1}^d(\alpha M_{ii}+\beta N_{ii}) =\alpha\sum_{i=1}^dM_{ii}+\beta\sum_{i=1}^dN_{ii}. \] Part 2 holds because transposing does not move the diagonal: \((\bfM^\top)_{ii}=M_{ii}\).

For part 3, write out both traces as double sums and swap the order of summation: \[ \operatorname{tr}(\bfB\bfC) =\sum_{i=1}^m(\bfB\bfC)_{ii} =\sum_{i=1}^m\sum_{j=1}^nB_{ij}C_{ji} =\sum_{j=1}^n\sum_{i=1}^mC_{ji}B_{ij} =\sum_{j=1}^n(\bfC\bfB)_{jj} =\operatorname{tr}(\bfC\bfB). \]

Part 4 is part 3 with the thin factors \(\bfB=\bfx\in\Rbb^{d\times1}\) and \(\bfC=\bfy^\top\in\Rbb^{1\times d}\): \[ \operatorname{tr}(\bfx\bfy^\top) =\operatorname{tr}(\bfy^\top\bfx) =\bfy^\top\bfx, \] since \(\bfy^\top\bfx\) is a \(1\times1\) matrix, and the trace of a \(1\times1\) matrix is its only entry.

Proposition 3 (The Frobenius norm and all the singular values) \[ \|\bfA\|_F=\left(\sum_{i=1}^r\sigma_i^2\right)^{1/2}. \]

Proof. Recall, the SVD of \(\bfA\) was formed from the spectral decomposition of \(\bfA^\top\bfA\). Hence, we can write \[ \bfA^\top\bfA =\sum_{i=1}^r\sigma_i^2\bfv_i\bfv_i^\top. \] Now take the trace, using linearity (Proposition 2, part 1) and then \(\operatorname{tr}(\bfv_i\bfv_i^\top)=\|\bfv_i\|_2^2=1\) (Proposition 2, part 4): \[ \|\bfA\|_F^2 =\operatorname{tr}(\bfA^\top\bfA) =\sum_{i=1}^r\sigma_i^2\operatorname{tr}(\bfv_i\bfv_i^\top) =\sum_{i=1}^r\sigma_i^2. \]

The cyclic property gives the same answer in one line, without expanding into rank-one pieces. Writing \(\bfA=\bfU_r\Sigma_r\bfV_r^\top\), we have \(\bfA^\top\bfA=\bfV_r\Sigma_r^2\bfV_r^\top\), and rotating \(\bfV_r\) to the back (Proposition 2, part 3) together with \(\bfV_r^\top\bfV_r=\bfI_r\) gives \[ \|\bfA\|_F^2 =\operatorname{tr}(\bfV_r\Sigma_r^2\bfV_r^\top) =\operatorname{tr}(\Sigma_r^2\bfV_r^\top\bfV_r) =\operatorname{tr}(\Sigma_r^2) =\sum_{i=1}^r\sigma_i^2. \]

So \(\|\bfA\|_2\) only sees the largest singular value, while \(\|\bfA\|_F\) sums the squares of all of them. This already explains why the two norms can behave so differently.

The same truncation is optimal in the Frobenius norm, and we take that on faith today.

Theorem 2 (Eckart-Young theorem, Frobenius norm) If \(k\) is an integer satisfying \(0\leq k<r\), then the truncated SVD \(\bfA_k\) is a best rank-at-most-\(k\) approximation of \(\bfA\) in the Frobenius norm: \[ \min_{\operatorname{rank}(\bfB)\leq k} \|\bfA-\bfB\|_F =\|\bfA-\bfA_k\|_F =\left(\sum_{i=k+1}^r\sigma_i^2\right)^{1/2}. \] For \(k\geq r\), the minimum is zero and is attained by \(\bfB=\bfA\).

Two remarks before we put it to work. The error is the tail of the singular values, exactly as in the spectral norm, but now the whole tail rather than its largest term (Proposition 3). And as in the spectral norm, the minimizer need not be unique: if \(\sigma_k=\sigma_{k+1}\), then there is a tie for which directions to keep.

Now we get to cash out geometrically.

Geometric Perspective: Orthogonal Projections

The centered data matrix

We switch to data notation. Everything proved above applies verbatim, with \(\bfX\) in place of \(\bfA\) and \((n,d)\) in place of \((m,n)\).

Definition 2 (Centered data matrix) Let \(\bfx_1,\ldots,\bfx_n\in\Rbb^d\) be observations, \(n\geq2\), with sample mean \[ \bar{\bfx}=\frac1n\sum_{i=1}^n\bfx_i\in\Rbb^d. \] The centered data matrix \(\bfX\in\Rbb^{n\times d}\) is the data matrix of the shifted observations: its \(i\)-th row is \((\bfx_i-\bar{\bfx})^\top\). Every column of \(\bfX\) has mean zero.

NoteStanding notation for the rest of the lecture

\(\bfX\in\Rbb^{n\times d}\) is centered, \(r=\operatorname{rank}(\bfX)\leq\min\{n,d\}\), and \[ \bfX=\bfU_r\Sigma_r\bfV_r^\top=\sum_{j=1}^r\sigma_j\bfu_j\bfv_j^\top, \qquad \sigma_1\geq\cdots\geq\sigma_r>0, \] with \(\bfu_j\in\Rbb^n\) and \(\bfv_j\in\Rbb^d\). For \(1\leq k\leq r\) we write \(\bfV_k=[\bfv_1\ \cdots\ \bfv_k]\in\Rbb^{d\times k}\), \(\bfU_k=[\bfu_1\ \cdots\ \bfu_k]\in\Rbb^{n\times k}\), and \(\bfX_k=\sum_{j=1}^k\sigma_j\bfu_j\bfv_j^\top\) for the truncated SVD of \(\bfX\). Note the observations live in \(\Rbb^d\), so the right singular vectors \(\bfv_j\) are the ones pointing in data space.

Subtracting \(\bar{\bfx}\) moves the origin to the center of the point cloud, so from here on every statement is about the variation about the mean rather than about the raw location of the data.

A reminder on orthogonal projection

Definition 3 (Orthogonal projection) Let \(S\subseteq\Rbb^d\) be a subspace with orthonormal basis \(\bfw_1,\ldots,\bfw_k\), and write \(\bfW=[\bfw_1\ \cdots\ \bfw_k]\in\Rbb^{d\times k}\), so that \(\bfW^\top\bfW=\bfI_k\). The orthogonal projection of \(\bfx\in\Rbb^d\) onto \(S\) is \[ \bfP_S\bfx =\sum_{i=1}^k(\bfw_i^\top\bfx)\bfw_i, \qquad \bfP_S=\bfW\bfW^\top\in\Rbb^{d\times d}. \] It keeps the component of \(\bfx\) in \(S\) and removes the component perpendicular to \(S\).

Theorem 3 (Pythagorean theorem) If \(\bfx,\bfy\in\Rbb^d\) are orthogonal, then \[ \|\bfx+\bfy\|_2^2=\|\bfx\|_2^2+\|\bfy\|_2^2. \]

Proof. Expand the squared norm and use \(\bfx^\top\bfy=0\): \[ \|\bfx+\bfy\|_2^2 =(\bfx+\bfy)^\top(\bfx+\bfy) =\|\bfx\|_2^2+2\bfx^\top\bfy+\|\bfy\|_2^2 =\|\bfx\|_2^2+\|\bfy\|_2^2. \]

Proposition 4 (Properties of orthogonal projection) Let \(S\subseteq\Rbb^d\) be a subspace of dimension \(k\), with \(\bfP_S=\bfW\bfW^\top\) as in Definition 3, and let \(\bfx\in\Rbb^d\). Then:

  1. \(\bfP_S^\top=\bfP_S\) and \(\bfP_S^2=\bfP_S\): projecting twice does the same thing as projecting once.
  2. \(\operatorname{rank}(\bfP_S)=k\), and \(\bfP_S\) does not depend on which orthonormal basis of \(S\) is used.
  3. \(\bfP_S\bfx\) is the unique closest point of \(S\) to \(\bfx\), that is, \(\|\bfx-\bfP_S\bfx\|_2<\|\bfx-\bfs\|_2\) for every \(\bfs\in S\) with \(\bfs\neq\bfP_S\bfx\).
  4. The residual \(\bfx-\bfP_S\bfx\) is perpendicular to \(S\), so by Theorem 3, \[ \|\bfx\|_2^2=\|\bfP_S\bfx\|_2^2+\|\bfx-\bfP_S\bfx\|_2^2. \]
  5. \(\bfI_d-\bfP_S\) is the orthogonal projection onto \(S^\perp\).

We use these without proof; see Section 6.3, Orthogonal Projection, of Interactive Linear Algebra.