Finishing the SVD Derivation, and Truncation

Math 123: Mathematical Aspects of Data Analysis – Fall 2026

Published

September 17, 2026

Cleaning up the SVD

Last time we set up the derivation of the singular value decomposition and stated, without proof, the lemma building the left singular vectors from the right ones. See the last set of lecture notes for the setup. Throughout, \(\bfA\in\Rbb^{m\times n}\) has rank \(r\), \(p=\min\{m,n\}\), and \(\bfv_1,\ldots,\bfv_n\) are the eigenvectors of \(\bfA^\top\bfA\) from the spectral theorem, with corresponding singular values \(\sigma_1\geq\cdots\geq\sigma_r>0=\sigma_{r+1}=\cdots=\sigma_p\) (singular values).

Lemma 1 (Building the left singular vectors) For \(i=1,\ldots,r\), define \[ \bfu_i=\frac{\bfA\bfv_i}{\sigma_i}\in\Rbb^m. \] Then:

\(\bfu_1,\ldots,\bfu_r\) are orthonormal;
\(\bfA\bfv_i=\sigma_i\bfu_i\) and \(\bfA^\top\bfu_i=\sigma_i\bfv_i\) for \(i\leq r\);
\(\bfA\bfv_i=\mathbf{0}\) for \(r<i\leq n\); and
\(\bfA\bfA^\top\bfu_i=\sigma_i^2\bfu_i\) for \(i\leq r\).

Proof

Proof. For \(i,j\leq r\), using \(\bfA^\top\bfA\bfv_j=\sigma_j^2\bfv_j\), \[ \bfu_i^\top\bfu_j =\frac{\bfv_i^\top\bfA^\top\bfA\bfv_j}{\sigma_i\sigma_j} =\frac{\sigma_j^2}{\sigma_i\sigma_j}\,\bfv_i^\top\bfv_j =\begin{cases} 1, & i=j,\\ 0, & i\neq j, \end{cases} \] since the \(\bfv_i\) are orthonormal. This proves 1.

The first identity in 2 is the definition of \(\bfu_i\). For the second, \[ \bfA^\top\bfu_i =\frac{\bfA^\top\bfA\bfv_i}{\sigma_i} =\frac{\sigma_i^2\bfv_i}{\sigma_i} =\sigma_i\bfv_i. \]

For 3, the computation above gives \(\|\bfA\bfv_i\|_2^2=\lambda_i=0\) when \(i>r\).

For 4, apply 2 twice: \(\bfA\bfA^\top\bfu_i=\bfA(\sigma_i\bfv_i)=\sigma_i\bfA\bfv_i=\sigma_i^2\bfu_i\).

Lemma 2 (Expansion in an orthonormal basis) Let \(\bfv_1,\ldots,\bfv_n\) be an orthonormal basis of \(\Rbb^n\). Then every \(\bfx\in\Rbb^n\) can be written as \[ \bfx=\sum_{i=1}^n c_i\bfv_i, \qquad c_i=\bfv_i^\top\bfx. \]

Proof

Proof. Since \(\bfv_1,\ldots,\bfv_n\) is a basis, there are scalars \(c_1,\ldots,c_n\) with \(\bfx=\sum_{i=1}^n c_i\bfv_i\). For each \(j=1,\ldots,n\), multiply on the left by \(\bfv_j^\top\) and use orthonormality: \[ \bfv_j^\top\bfx =\sum_{i=1}^n c_i\,\bfv_j^\top\bfv_i =c_j, \] since \(\bfv_j^\top\bfv_i=0\) for \(i\neq j\) and \(\bfv_j^\top\bfv_j=1\).

Theorem 1 (Singular value decomposition) Every matrix \(\bfA\in\Rbb^{m\times n}\) of rank \(r\) admits a decomposition \[ \bfA=\bfU_r\Sigma_r\bfV_r^\top =\sum_{i=1}^r\sigma_i\bfu_i\bfv_i^\top, \] where

\(\bfU_r=[\bfu_1\ \cdots\ \bfu_r]\in\Rbb^{m\times r}\) has orthonormal columns, that is, \(\bfU_r^\top\bfU_r=\bfI_r\);
\(\bfV_r=[\bfv_1\ \cdots\ \bfv_r]\in\Rbb^{n\times r}\) has orthonormal columns, that is, \(\bfV_r^\top\bfV_r=\bfI_r\); and
\(\Sigma_r=\operatorname{diag}(\sigma_1,\ldots,\sigma_r)\in\Rbb^{r\times r}\), with \(\sigma_1\geq\cdots\geq\sigma_r>0\).

Proof

Proof. Take \(\bfv_1,\ldots,\bfv_n\) and \(\sigma_1,\ldots,\sigma_r\) from the setup above, and \(\bfu_1,\ldots,\bfu_r\) from Lemma 1. Since \(\bfv_1,\ldots,\bfv_n\) is an orthonormal basis of \(\Rbb^n\), every \(\bfx\in\Rbb^n\) can be written \(\bfx=\sum_{i=1}^n c_i\bfv_i\) with \(c_i=\bfv_i^\top\bfx\) (Lemma 2). Then \[ \bfA\bfx =\sum_{i=1}^n c_i\bfA\bfv_i =\sum_{i=1}^r c_i\bfA\bfv_i =\sum_{i=1}^r \sigma_i c_i\bfu_i =\left(\sum_{i=1}^r\sigma_i\bfu_i\bfv_i^\top\right)\bfx, \] using \(\bfA\bfv_i=\mathbf{0}\) for \(i>r\) and \(\bfA\bfv_i=\sigma_i\bfu_i\) for \(i\leq r\) (Lemma 1). Since this holds for every \(\bfx\in\Rbb^n\), \[ \bfA=\sum_{i=1}^r\sigma_i\bfu_i\bfv_i^\top =\bfU_r\Sigma_r\bfV_r^\top. \]

This is exactly the decomposition the greyscale image example asked for. Each term \(\sigma_i\bfu_i\bfv_i^\top\) is a rank-one image: \(\bfu_i\) is a unit-length column profile, \(\bfv_i\) is a unit-length row profile, and \(\sigma_i\) says how much that piece contributes; as a linear map, \(\bfu_i\bfv_i^\top\bfx=(\bfv_i^\top\bfx)\bfu_i\).

Definition 1 (Singular vectors) Let \(\bfA=\bfU_r\Sigma_r\bfV_r^\top\) be as in Theorem 1. The columns \(\bfv_1,\ldots,\bfv_r\) of \(\bfV_r\) are right singular vectors of \(\bfA\), and the columns \(\bfu_1,\ldots,\bfu_r\) of \(\bfU_r\) are left singular vectors of \(\bfA\).

Reading Theorem 1 (or Lemma 1) shows that singular vectors come in matched pairs: \[ \bfA\bfv_i=\sigma_i\bfu_i, \qquad \bfA^\top\bfu_i=\sigma_i\bfv_i, \qquad 1\leq i\leq r. \]

The matrices \(\bfU_r\) and \(\bfV_r\) above are rectangular (unless \(m=n=r\)), so Theorem 1 is often called the compact or rank-\(r\) SVD. It is all we will ever need, but it is worth knowing that a version with square orthogonal matrices also exists, since that version is sometimes more convenient to reason with (e.g. for orthogonality arguments).

Theorem 2 (Full singular value decomposition) Every matrix \(\bfA\in\Rbb^{m\times n}\) of rank \(r\) admits a full singular value decomposition \[ \bfA=\bfU\Sigma\bfV^\top, \] where

\(\bfU\in\Rbb^{m\times m}\) is orthogonal, that is, \(\bfU^\top\bfU=\bfU\bfU^\top=\bfI_m\);
\(\bfV\in\Rbb^{n\times n}\) is orthogonal, that is, \(\bfV^\top\bfV=\bfV\bfV^\top=\bfI_n\); and
\(\Sigma\in\Rbb^{m\times n}\) is zero off its main diagonal, with diagonal entries the singular values \[ \sigma_1\geq\sigma_2\geq\cdots\geq \sigma_p\geq0, \qquad p=\min\{m,n\}. \]

Exactly \(r\) singular values are positive, and the remaining \(p-r\) are zero.

For a data matrix with \(n=10^6\) observations and \(d=10^4\) features, the full \(\bfU\) is \(10^6\times10^6\) and the full \(\bfV\) is \(10^4\times10^4\), while the compact \(\bfU_r,\bfV_r\) have only \(r\leq10^4\) columns each. This is why, from here on, we work directly with the compact SVD \(\bfU_r\Sigma_r\bfV_r^\top\).

Proposition 1 (The SVD and eigendecomposition) Let \(\bfA\in\Rbb^{m\times n}\) have compact SVD \(\bfA=\bfU_r\Sigma_r\bfV_r^\top\) as in Theorem 1. Then \[ \bfA^\top\bfA =\bfV_r\Sigma_r^2\bfV_r^\top, \qquad \bfA\bfA^\top =\bfU_r\Sigma_r^2\bfU_r^\top. \] In particular, \(\bfA^\top\bfA\bfv_i=\sigma_i^2\bfv_i\) and \(\bfA\bfA^\top\bfu_i=\sigma_i^2\bfu_i\) for \(1\leq i\leq r\), and every other eigenvalue of \(\bfA^\top\bfA\) and \(\bfA\bfA^\top\) is zero.

Proof

Proof. Substitute: \(\bfA^\top\bfA=\bfV_r\Sigma_r^\top\bfU_r^\top\bfU_r\Sigma_r\bfV_r^\top=\bfV_r\Sigma_r^2\bfV_r^\top\), using \(\bfU_r^\top\bfU_r=\bfI_r\); the computation for \(\bfA\bfA^\top\) is identical, using \(\bfV_r^\top\bfV_r=\bfI_r\). Since \(\bfU_r\) has orthonormal columns and \(\Sigma_r^2\) is an invertible \(r\times r\) diagonal matrix, \(\bfA\bfA^\top=\bfU_r\Sigma_r^2\bfU_r^\top\) has rank \(r\), so it has exactly \(r\) nonzero eigenvalues, namely \(\sigma_1^2,\ldots,\sigma_r^2\) with eigenvectors \(\bfu_1,\ldots,\bfu_r\); every other eigenvalue of \(\bfA\bfA^\top\) is zero. The same argument applies to \(\bfA^\top\bfA=\bfV_r\Sigma_r^2\bfV_r^\top\).

Singular vectors are not unique

Replacing \((\bfu_1,\bfv_1)\) by \((-\bfu_1,-\bfv_1)\) gives the same term \(\sigma_1\bfu_1\bfv_1^\top\), so singular vectors are at best determined up to sign. When an eigenvalue of \(\bfA^\top\bfA\) is repeated (like the zero eigenvalue above, with eigenspace spanned by the vectors orthogonal to \(\bfw\)), any orthonormal basis of the corresponding eigenspace works, though only eigenvectors with \(\sigma_i>0\) ever become singular vectors of \(\bfA\) (Definition 1). What is determined is the list of singular values and the subspaces.

This is also why the proof of Theorem 1 built \(\bfu_i\) from \(\bfv_i\) instead of eigendecomposing \(\bfA\bfA^\top\) separately: independently chosen unit eigenvectors of \(\bfA^\top\bfA\) and \(\bfA\bfA^\top\) need not satisfy \(\bfA\bfv_i=\sigma_i\bfu_i\) (flip the sign of just one of them).

Truncating & Low Rank Approximation

Since the singular values are sorted, the natural way to approximate \(\bfA\) is to keep only the first few terms of the rank-one sum.

Definition 2 (Truncated singular value decomposition) Let \(\bfA\) have rank \(r\). For an integer \(k\) satisfying \(0\leq k\leq r\), its rank-\(k\) truncated SVD is \[ \bfA_k =\sum_{i=1}^k\sigma_i\bfu_i\bfv_i^\top =\bfU_k\Sigma_k\bfV_k^\top, \] with \(\bfA_0=\mathbf{0}\). For \(1\leq k\leq r\), \(\operatorname{rank}(\bfA_k)=k\).

Storing \(\bfA_k\) requires \(k(m+n+1)\) numbers instead of \(mn\). For a photograph, the singular values typically decay quickly: a few are large and most are small. Then \(\bfA_k\) with modest \(k\) keeps the large terms and throws away the small ones, which is the idea behind SVD-based image compression.

This answers question 1 from the start of lecture, and raises question 2 again in sharper form: is \(\bfA_k\) the best rank-\(k\) approximation of \(\bfA\), and how large is the error? The answer is the Eckart–Young theorem: the error is controlled exactly by the discarded singular values \(\sigma_{k+1},\sigma_{k+2},\ldots\).

To say that \(\bfA_k\) is best, we first need a way to measure the size of the error matrix \(\bfA-\bfB\). Two norms are standard. The first ignores that a matrix is a linear map and simply treats it as a long vector of entries.

Definition 3 (Frobenius norm) The trace of a square matrix \(\bfC\in\Rbb^{d\times d}\) is the sum of its diagonal entries: \[ \operatorname{tr}(\bfC)=\sum_{\ell=1}^d C_{\ell\ell}. \] For \(\bfM,\bfN\in\Rbb^{m\times n}\), the Frobenius inner product is \[ \langle\bfM,\bfN\rangle_F =\operatorname{tr}(\bfM^\top\bfN) =\sum_{i=1}^m\sum_{j=1}^n M_{ij}N_{ij}. \] The corresponding Frobenius norm is \[ \|\bfM\|_F =\sqrt{\langle\bfM,\bfM\rangle_F} =\sqrt{\operatorname{tr}(\bfM^\top\bfM)} =\left(\sum_{i=1}^m\sum_{j=1}^n M_{ij}^2\right)^{1/2}. \]

That is, \(\|\bfM\|_F\) is the Euclidean norm of \(\bfM\) reshaped into a vector in \(\Rbb^{mn}\). For an image, \(\|\bfA-\bfB\|_F^2\) is the total squared pixel error. The second norm instead measures what \(\bfM\) does to vectors.

Definition 4 (Spectral norm) For \(\bfM\in\Rbb^{m\times n}\), the spectral norm is \[ \|\bfM\|_2 =\max_{\|\bfx\|_2=1}\|\bfM\bfx\|_2. \]

The spectral norm is the largest factor by which \(\bfM\) can stretch a vector. Equivalently, it is the smallest constant \(C\) with \[ \|\bfM\bfx\|_2\leq C\|\bfx\|_2 \qquad\text{for all }\bfx\in\Rbb^n. \] Some intuition on the idea is in the setting of worst case error. If \(\|\bfA-\bfB\|_2\leq\varepsilon\), then \(\|\bfA\bfx-\bfB\bfx\|_2\leq\varepsilon\|\bfx\|_2\) for every input \(\bfx\). A small spectral error is a uniform guarantee that \(\bfB\) acts like \(\bfA\), whereas a small Frobenius error only says the entries are close on average.

The two norms can differ considerably: \(\|\bfI_n\|_2=1\), while \(\|\bfI_n\|_F=\sqrt{n}\). In general \(\|\bfM\|_2\leq\|\bfM\|_F\leq\sqrt{\operatorname{rank}(\bfM)}\,\|\bfM\|_2\).

Theorem 3 (Eckart–Young theorem) Let \(\bfA\in\Rbb^{m\times n}\) have rank \(r\) and singular values \(\sigma_1\geq\cdots\geq\sigma_r>0\). If \(k\) is an integer satisfying \(0\leq k<r\), then the truncated SVD \(\bfA_k\) is a best rank-at-most-\(k\) approximation of \(\bfA\) in the spectral norm: \[ \min_{\operatorname{rank}(\bfB)\leq k} \|\bfA-\bfB\|_2 =\|\bfA-\bfA_k\|_2 =\sigma_{k+1}. \] When \(0<k<r\), a minimizing matrix need not be unique. For \(k\geq r\), the minimum is zero and is attained by \(\bfB=\bfA\).

The same truncation is also optimal in the Frobenius norm.

Theorem 4 (Eckart–Young theorem, Frobenius norm) With \(\bfA\), \(r\), and \(k\) as in Theorem 3, \[ \min_{\operatorname{rank}(\bfB)\leq k} \|\bfA-\bfB\|_F =\|\bfA-\bfA_k\|_F =\left(\sum_{i=k+1}^r\sigma_i^2\right)^{1/2}. \]

So the singular values are key components in both building the approximation and in reporting its error. A quickly decaying sequence \(\sigma_1,\sigma_2,\ldots\) means a small \(k\) already captures almost all of \(\bfA\).