Finishing up Caratheodory, and then the SVD

Math 123: Mathematical Aspects of Data Analysis – Fall 2026

Finishing up Caratheodory

I need to finish up proving this darn thing! See the last set of lecture notes to refresh yourself on the problem.

Theorem 1 (Approximate Carathéodory theorem) Let \(\mathcal{V}=\{\bfv_1,\ldots,\bfv_n\}\subseteq\Rbb^d\) satisfy \(\|\bfv_i\|_2\leq R\) for every \(i\), and let \(\bfx\in\operatorname{conv}(\mathcal{V})\). For every positive integer \(k\), there are \(k\) points \(\bfu_1,\ldots,\bfu_k\in\mathcal{V}\), possibly with repetitions, such that \[ \left\| \bfx-\frac1k\sum_{j=1}^k\bfu_j \right\|_2 \leq \frac{R}{\sqrt{k}}. \] Consequently, for every \(\epsilon>0\), there is a convex combination of at most \[ \max\left\{1,\left\lceil\frac{R^2}{\epsilon^2}\right\rceil\right\} \] points of \(\mathcal{V}\) whose Euclidean distance from \(\bfx\) is at most \(\epsilon\).

NoteProof

Proof. Because \(\bfx\in\operatorname{conv}(\mathcal{V})\), choose non-negative weights \(\alpha_1,\ldots,\alpha_n\) whose sum is \(1\) and for which \[ \bfx=\sum_{i=1}^n\alpha_i\bfv_i. \] Define a random vector \(\bfX\) by sampling from this convex combination: \[ \Pbb(\bfX=\bfv_i)=\alpha_i. \] Then \(\Ebb[\bfX]=\bfx\). Take \(k\) independent copies \(\bfX_1,\ldots,\bfX_k\) of \(\bfX\) and form their average \[ \hat{\bfx}=\frac1k\sum_{j=1}^k\bfX_j. \] Every possible value of \(\hat{\bfx}\) is an average of \(k\) points from \(\mathcal{V}\). Moreover, independence and \(\Ebb[\bfX_j-\bfx]=\mathbf{0}\) make all of the cross terms vanish. Indeed, if \(j\neq \ell\), then \[ \Ebb\left[ \left\langle \bfX_j-\bfx,\bfX_\ell-\bfx\right\rangle \right] = \left\langle \Ebb[\bfX_j-\bfx],\Ebb[\bfX_\ell-\bfx] \right\rangle =0. \] It follows that \[ \begin{aligned} \Ebb\left[\|\hat{\bfx}-\bfx\|_2^2\right] &=\frac{1}{k^2} \Ebb\left[ \left\|\sum_{j=1}^k(\bfX_j-\bfx)\right\|_2^2 \right] \\ &=\frac{1}{k}\Ebb\left[\|\bfX-\bfx\|_2^2\right] \\ &=\frac{1}{k}\left( \Ebb\left[\|\bfX\|_2^2\right]-\|\bfx\|_2^2 \right) \\ &\leq\frac{R^2}{k}. \end{aligned} \] The squared error is non-negative, so at least one outcome is no larger than its expectation. For that outcome, \(\|\hat{\bfx}-\bfx\|_2\leq R/\sqrt{k}\). Fixing it proves the first claim. Taking \(k=\max\{1,\lceil R^2/\epsilon^2\rceil\}\) proves the second.

For example, if the data are normalized so that \(R=1\), then \(100\) points suffice for error \(0.1\) whether the ambient dimension is \(100\), \(10{,}000\), or one million.

Thus a high-dimensional convex representation can contain substantial redundancy. Once a controlled amount of error is acceptable, a simple random sampling argument produces a sparse approximation whose size is controlled by the desired accuracy, not by the ambient dimension.

Looking for Lower-Dimensional Structure

I presented Carathéodory because it’s very illustrative, albeit in a limited setting. Accepting tolerable error, there often may exist a notion of redundancy in high dimensional data. Let’s return to our beloved data matrix and ask the linear algebraic question: How do I understand this idea here?

Example 1 (Compressing a greyscale image) A greyscale image with \(m\) rows and \(n\) columns of pixels is a matrix \(\bfA\in\Rbb^{m\times n}\), where \(A_{ij}\) is the brightness of pixel \((i,j)\) (say \(0\) is black and larger values are brighter). A \(3000\times4000\) photo is \(12\) million numbers.

Here is a tiny \(3\times3\) image. Let \(\bfw=(1,2,3)^\top\) and \[ \bfA=\bfw\bfw^\top = \begin{pmatrix} 1 & 2 & 3 \\ 2 & 4 & 6 \\ 3 & 6 & 9 \end{pmatrix}. \] It is darkest in the top-left corner and brightens toward the bottom-right. There are \(9\) pixels, but row \(i\) is just \(w_i\bfw^\top\), so every row is a multiple of the same row and \(\operatorname{rank}(\bfA)=1\). The whole image is described by the \(3\) numbers in \(\bfw\).

More generally, any rank-one image \(\bfA=\bfu\bfv^\top\) with \(\bfu\in\Rbb^m\) and \(\bfv\in\Rbb^n\) is a “column profile times a row profile,” and can be stored with \(m+n\) numbers instead of \(mn\). A sum of \(k\) such images costs \(k(m+n)\) numbers. For a \(3000\times4000\) image with \(k=50\), that is \(350{,}000\) numbers instead of \(12{,}000{,}000\), about \(3\%\).

Of course, a real photograph is not a single outer product. But natural images are highly structured: neighboring rows look alike, and large regions are smooth. So we might hope that \[ \bfA\approx\sum_{i=1}^k\bfu_i\bfv_i^\top, \qquad k\ll\min\{m,n\}. \] The same hope drives much of data science: users-by-movies rating matrices, genes-by-samples expression matrices, and collections of face images are all approximately described by a few rank-one pieces.

However, real data are never exactly low rank, and rank is fragile. Change the top-left pixel of the image above from \(1\) to \(1.01\) and the rank jumps to \(2\); add a little noise everywhere and it jumps to \(3\). So we are left with two questions:

1. Given only \(\bfA\), how do we discover the rank-one pieces?
2. When \(\bfA\) is only approximately low rank, what is the best low-rank approximation, and how good is it?

The singular value decomposition answers both.

Linear Algebra Review

Definition 1 (Data matrix) A dataset with \(n\) observations and \(d\) features is stored as a matrix \[ \bfX = \begin{pmatrix} - & \bfx_1^\top & - \\ - & \bfx_2^\top & - \\ & \vdots & \\ - & \bfx_n^\top & - \end{pmatrix} \in \Rbb^{n\times d}. \] Row \(i\) is the observation \(\bfx_i^\top\), and column \(j\) is the \(j\)-th feature across all observations.

Definition 2 (Eigenvalue and eigenvector) Let \(\bfA\in\Rbb^{n\times n}\). A scalar \(\lambda\in\Rbb\) is an eigenvalue of \(\bfA\) if there exists a nonzero vector \(\bfv\in\Rbb^n\) such that \[ \bfA\bfv=\lambda\bfv. \] The vector \(\bfv\) is an eigenvector of \(\bfA\) corresponding to \(\lambda\).

Theorem 2 (Spectral theorem) Let \(\bfA\in\Rbb^{n\times n}\) be symmetric. There exist an orthogonal matrix \(\bfQ\in\Rbb^{n\times n}\) and a diagonal matrix \[ \Lambda=\operatorname{diag}(\lambda_1,\ldots,\lambda_n), \qquad \lambda_1\geq\cdots\geq\lambda_n, \] such that \[ \bfA=\bfQ\Lambda\bfQ^\top =\sum_{i=1}^n\lambda_i\bfq_i\bfq_i^\top. \] The eigenvalues \(\lambda_1,\ldots,\lambda_n\) are real, and the columns \(\bfq_1,\ldots,\bfq_n\) of \(\bfQ\) form an orthonormal basis of eigenvectors of \(\bfA\).

However, a data matrix is rectangular, so it has no eigenvalues at all. Is there some other matrix we can look at to understand a notion similar to the spectrum of \(\bfA\)? Enter the square matrices \(\bfA^\top\bfA\in\Rbb^{n\times n}\) and \(\bfA\bfA^\top\in\Rbb^{m\times m}\). Both are symmetric, so Theorem 2 applies to them. They’re also postiive semidefinite which, recall, is the following property.

Definition 3 (Positive semidefinite matrix) A symmetric matrix \(\bfM\in\Rbb^{n\times n}\) is positive semidefinite (PSD) if \[ \bfx^\top\bfM\bfx\geq0 \qquad\text{for every }\bfx\in\Rbb^n. \] If moreover \(\bfx^\top\bfM\bfx>0\) for every nonzero \(\bfx\), then \(\bfM\) is positive definite.

For example, \(\bfI_n\) is positive definite, since \(\bfx^\top\bfI_n\bfx=\|\bfx\|_2^2\), while \(\operatorname{diag}(1,-1)\) is not PSD, since \(\bfe_2^\top\operatorname{diag}(1,-1)\bfe_2=-1\). For symmetric matrices, being PSD is exactly a statement about the eigenvalues.

Proposition 1 (PSD matrices have non-negative eigenvalues) A symmetric matrix \(\bfM\in\Rbb^{n\times n}\) is PSD if and only if every eigenvalue of \(\bfM\) is non-negative.

NoteProof

Proof. Suppose \(\bfM\) is PSD, and let \(\bfM\bfv=\lambda\bfv\) with \(\|\bfv\|_2=1\). Then \[ \lambda=\lambda\bfv^\top\bfv=\bfv^\top\bfM\bfv\geq0. \] Conversely, suppose every eigenvalue is non-negative, and write \(\bfM=\bfQ\Lambda\bfQ^\top\) as in Theorem 2. For any \(\bfx\in\Rbb^n\), set \(\bfy=\bfQ^\top\bfx\). Then \[ \bfx^\top\bfM\bfx =\bfy^\top\Lambda\bfy =\sum_{i=1}^n\lambda_i y_i^2 \geq0. \]

Proposition 2 (\(\bfA^\top\bfA\) and \(\bfA\bfA^\top\) are PSD) Let \(\bfA\in\Rbb^{m\times n}\). Then \(\bfA^\top\bfA\) and \(\bfA\bfA^\top\) are PSD. Moreover, \[ \operatorname{null}(\bfA^\top\bfA)=\operatorname{null}(\bfA), \qquad\text{so}\qquad \operatorname{rank}(\bfA^\top\bfA)=\operatorname{rank}(\bfA). \]

You will prove Proposition 2 in Homework 2.

The Singular Value Decomposition

We now derive the SVD. The outline of the idea is:

1. Eigendecompose the PSD matrix \(\bfA^\top\bfA\). This produces \(\bfV\) and the singular values.
2. Push the eigenvectors \(\bfv_i\) through \(\bfA\) and normalize. This produces \(\bfU\).
3. Check that the pieces fit together as \(\bfA=\bfU\Sigma\bfV^\top\).

Throughout, fix \(\bfA\in\Rbb^{m\times n}\) with rank \(r\), and let \(p=\min\{m,n\}\). Note that \(r\leq p\).

By Proposition 2, \(\bfA^\top\bfA\) is symmetric, so Theorem 2 gives an orthogonal \(\bfV=[\bfv_1\ \cdots\ \bfv_n]\in\Rbb^{n\times n}\) with \[ \bfA^\top\bfA=\bfV\Lambda\bfV^\top, \qquad \Lambda=\operatorname{diag}(\lambda_1,\ldots,\lambda_n), \qquad \bfA^\top\bfA\bfv_i=\lambda_i\bfv_i. \] By Proposition 1, every \(\lambda_i\geq0\). Since \(\bfV\) is invertible, \(\operatorname{rank}(\bfA^\top\bfA)=\operatorname{rank}(\Lambda)\), the number of nonzero \(\lambda_i\), and by Proposition 2 this number is \(r\). Therefore \[ \lambda_1\geq\cdots\geq\lambda_r>0=\lambda_{r+1}=\cdots=\lambda_n. \]

Definition 4 (Singular values) The singular values of \(\bfA\) are \[ \sigma_i=\sqrt{\lambda_i}, \qquad i=1,\ldots,p, \] where \(\lambda_1\geq\cdots\geq\lambda_n\geq0\) are the eigenvalues of \(\bfA^\top\bfA\). Thus \[ \sigma_1\geq\cdots\geq\sigma_r>0=\sigma_{r+1}=\cdots=\sigma_p. \]

The eigenvalues of \(\bfA^\top\bfA\) do not depend on which eigenvectors we picked, so the singular values are determined by \(\bfA\) alone.

Now, notice, the vector \(\bfA\bfv_i\) has length exactly \(\sigma_i\): \[ \|\bfA\bfv_i\|_2^2 =\bfv_i^\top\bfA^\top\bfA\bfv_i =\lambda_i\bfv_i^\top\bfv_i =\sigma_i^2. \] So \(\bfA\) sends \(\bfv_i\) to a vector of length \(\sigma_i\), which is nonzero exactly when \(i\leq r\). Normalizing these images gives the other half of the decomposition.

Lemma 1 (Building the left singular vectors) For \(i=1,\ldots,r\), define \[ \bfu_i=\frac{\bfA\bfv_i}{\sigma_i}\in\Rbb^m. \] Then:

1. \(\bfu_1,\ldots,\bfu_r\) are orthonormal;
2. \(\bfA\bfv_i=\sigma_i\bfu_i\) and \(\bfA^\top\bfu_i=\sigma_i\bfv_i\) for \(i\leq r\);
3. \(\bfA\bfv_i=\mathbf{0}\) for \(r<i\leq n\); and
4. \(\bfA\bfA^\top\bfu_i=\sigma_i^2\bfu_i\) for \(i\leq r\).

We will prove this lemma, and build the rest of the SVD from it, next time.