Spectral embeddings and spectral clustering
Math 123: Mathematical Aspects of Data Analysis – Fall 2026
Where we are
Last time we turned a point cloud into a weighted graph and met its Laplacian \(\bfL=\bfD-\bfW\). Its quadratic form measures roughness, \(\bff^\top\bfL\bff=\frac12\sum_{i,j}w_{ij}(f_i-f_j)^2\), and on indicator vectors it measures cuts, \(\mathbf{1}_S^\top\bfL\mathbf{1}_S=\operatorname{cut}(S)\). We also stated Rayleigh–Ritz: eigenvectors of symmetric matrices solve optimization problems.
Today we first see that the functions with zero roughness are exactly those constant on each connected component. Then we use Rayleigh–Ritz to split a graph in two by relax and round: write the cut problem as a quadratic form, drop the requirement that the vector take only two values, solve the relaxed problem with an eigenvector, and round the eigenvector back to a split. Keeping several eigenvectors instead of one gives the smoothest ways to place the vertices in \(\Rbb^k\), and \(k\)-means in that embedding finds \(k\) clusters. Spectral clustering is one use of that idea, but the same eigenvectors are the tool behind the nonlinear dimension reduction methods of Chapter 6, and they reappear in modern graph neural networks and self-supervised learning.
This page covers part of Sections 5.4–5.5 of the textbook: spectral clustering with the Laplacian \(\bfL\) itself. The version that weights each vertex by its degree, the normalized cut, and Cheeger’s inequality are in Lecture 16. For a nice alternative presentation, see the tutorial of von Luxburg (Luxburg 2007).
Throughout, \(\mathbf{1}\in\Rbb^n\) is the all-ones vector, \(\mathbf{1}_S\in\Rbb^n\) is the indicator vector of a set \(S\) of vertices.
Smooth functions see connected components
The functions with zero roughness see exactly the connected components.
Proof. If \(\bfL\bff=\mathbf{0}\), then \(\bff^\top\bfL\bff=0\), so by the Laplacian quadratic form every term \(w_{ij}(f_i-f_j)^2\) vanishes and \(f_i=f_j\) across every edge. Following walks, \(\bff\) is constant on each component, so \(\bff\) is a combination of the \(\mathbf{1}_{C_\ell}\).
Conversely, fix a component \(C_\ell\) and a vertex \(i\). By the definition of \(\bfL\), \((\bfL\mathbf{1}_{C_\ell})_i=\deg(i)(\mathbf{1}_{C_\ell})_i-\sum_{j\in C_\ell}w_{ij}\). If \(i\in C_\ell\), every neighbour of \(i\) also lies in \(C_\ell\), so \(\sum_{j\in C_\ell}w_{ij}=\sum_{j=1}^nw_{ij}=\deg(i)\) and the entry is \(\deg(i)-\deg(i)=0\). If \(i\notin C_\ell\), then \((\mathbf{1}_{C_\ell})_i=0\), and \(i\) has no neighbours in \(C_\ell\), so both terms vanish. Hence \(\mathbf{1}_{C_\ell}\in\ker\bfL\) for every \(\ell\).
Finally, the indicators have disjoint supports, so they are linearly independent and \(\dim\ker\bfL=k\).
So the graph is connected exactly when \(\mathbf{0}\) is a simple eigenvalue of \(\bfL\). The guiding heuristic for the rest of the unit: a graph that is almost disconnected should have almost a second zero eigenvalue, with an eigenvector that is almost constant on each piece.
Rayleigh–Ritz: eigenvectors as optimizers
To make this heuristic precise, we need the fact, stated last time, that eigenvectors of symmetric matrices solve optimization problems. We have used one instance already: the top right singular vectors span the best-fitting subspace. Last time we stated it with the largest eigenvalues, which suits PCA. On graphs we want the opposite, the smoothest vectors, which minimize roughness, so we need the smallest eigenvalues. Here is the statement in that form, which we will use now for graphs and, in one form or the other, for every embedding method in Chapter 6.
In class we prove only the case \(m=1\), which is all we need for splitting a graph in two. The proof of the general case is included below for reference.
Proof (Proof for a single vector). Let \(\|\bfz\|_2=1\) and expand \(\bfz=\sum_jc_j\bfu_j\) in the eigenbasis, with \(c_j=\bfu_j^\top\bfz\). Since the \(\bfu_j\) are orthonormal, \(\sum_jc_j^2=\|\bfz\|_2^2=1\), and since \(\bfB\bfz=\sum_j\lambda_jc_j\bfu_j\), \[ \bfz^\top\bfB\bfz=\sum_{j=1}^n\lambda_jc_j^2\geq\lambda_1\sum_{j=1}^nc_j^2=\lambda_1. \] In words, \(\bfz^\top\bfB\bfz\) is a weighted average of the eigenvalues with weights \(c_j^2\), and an average is at least its smallest term. If moreover \(\bfz\perp\bfu_1\), then \(c_1=0\), the sums run over \(j\geq2\) only, and the same bound gives \(\bfz^\top\bfB\bfz\geq\lambda_2\). Equality holds at \(\bfz=\bfu_1\) (respectively \(\bfz=\bfu_2\)), since then \(\bfz^\top\bfB\bfz=\lambda_1\bfu_1^\top\bfu_1=\lambda_1\) (respectively \(\lambda_2\)).
Proof (Proof of the general case (not covered in class)). By the spectral theorem, \(\bfB=\sum_j\lambda_j\bfu_j\bfu_j^\top\), so \[ \operatorname{tr}(\bfZ^\top\bfB\bfZ)=\sum_{j=1}^n\lambda_jp_j, \qquad p_j=\|\bfZ^\top\bfu_j\|_2^2. \] For \(m=1\) these weights are the \(c_j^2\) of the single-vector proof. They sum to \(1\) there, so each is automatically at most \(1\); for general \(m\) we have to check this separately. The weights satisfy \(0\leq p_j\leq1\), because \(\|\bfZ^\top\bfu_j\|_2=\|\bfZ\bfZ^\top\bfu_j\|_2\) is the length of the orthogonal projection of the unit vector \(\bfu_j\) onto the column space of \(\bfZ\). They also satisfy \(\sum_jp_j=\|\bfZ^\top\bfU\|_F^2=\|\bfZ\|_F^2=m\), where \(\bfU=[\bfu_1,\ldots,\bfu_n]\) is orthogonal. For \(j\leq m\) we have \(\lambda_j\leq\lambda_m\) and \(p_j-1\leq0\), while for \(j>m\) we have \(\lambda_j\geq\lambda_m\) and \(p_j\geq0\). Hence \[ \sum_j\lambda_jp_j-\sum_{j\leq m}\lambda_j =\sum_{j\leq m}\lambda_j(p_j-1)+\sum_{j>m}\lambda_jp_j \geq\lambda_m\sum_{j\leq m}(p_j-1)+\lambda_m\sum_{j>m}p_j =\lambda_m\Big(\sum_jp_j-m\Big)=0. \] If \(\bfZ^\top\bfu_1=\mathbf{0}\), then \(p_1=0\); run the same argument over \(j\geq2\), with \(\lambda_{m+1}\) in place of \(\lambda_m\). The equality cases are direct computations.
Flipping signs, the largest eigenvalues and their eigenvectors solve the corresponding maximization problems. This is the form we stated last time.
This follows by applying Proposition 2 to \(-\bfB\), which has the same eigenvectors and eigenvalues \(-\beta_1\leq\cdots\leq-\beta_n\) in increasing order. The two results bound the same trace from opposite sides: in the single ordering \(\lambda_1\leq\cdots\leq\lambda_n\), every \(\bfZ\) with \(\bfZ^\top\bfZ=\bfI_m\) satisfies \[ \lambda_1+\cdots+\lambda_m\leq\operatorname{tr}(\bfZ^\top\bfB\bfZ)\leq\lambda_{n-m+1}+\cdots+\lambda_n. \] PCA uses the maximization (directions of most variance); graphs will use the minimization (directions of least roughness).
Splitting a graph in two
Last time we saw that \(\mathbf{1}_S^\top\bfL\mathbf{1}_S=\operatorname{cut}(S)\), so finding a set with a small cut means finding a smooth \(0/1\) vector. Minimizing the cut alone fails: \(S=\emptyset\) and \(S=V\) have cut \(0\). In vector terms, the minimizers are the constant vectors, which by Proposition 1 are exactly the zero-roughness vectors of a connected graph. So we have to rule out the constants, and also lopsided splits such as cutting off a single vertex of low degree.
The simplest fix is to ask for a balanced split. Suppose \(n\) is even and require \(|S|=n/2\). Encode \(S\) by signs instead of \(0\)s and \(1\)s: \[ \bfy=\mathbf{1}_S-\mathbf{1}_{S^c}, \qquad y_i=\begin{cases}+1&i\in S,\\-1&i\notin S.\end{cases} \] Then \((y_i-y_j)^2\) is \(4\) on edges crossing the cut and \(0\) on all other edges, so \(\bfy^\top\bfL\bfy=4\operatorname{cut}(S)\). Balance is the condition \(\bfy^\top\mathbf{1}=|S|-|S^c|=0\), and every sign vector has \(\|\bfy\|_2^2=n\). Hence \[ \min_{|S|=n/2}\operatorname{cut}(S) =\frac14\min\big\{\bfy^\top\bfL\bfy:\ \bfy\in\{\pm1\}^n,\ \bfy\perp\mathbf{1}\big\}. \] Nothing has changed yet: this is the same problem, written as a quadratic form. It is still a search over exponentially many sets, and finding the minimum balanced cut is NP-hard in general.
Now relax: keep \(\|\bfy\|_2^2=n\) and \(\bfy\perp\mathbf{1}\), but allow any real entries.
Proof. By Proposition 1, \(\ker\bfL\) is spanned by \(\mathbf{1}\), so \(\lambda_1=0\) is simple and we may take \(\bfu_1=\mathbf{1}/\sqrt n\). Write \(\bfy=\sqrt n\,\bfz\). The constraints become \(\|\bfz\|_2=1\) and \(\bfz\perp\bfu_1\), and \(\bfy^\top\bfL\bfy=n\,\bfz^\top\bfL\bfz\). By Proposition 2 with \(m=1\), \(\bfz^\top\bfL\bfz\geq\lambda_2\), with equality at \(\bfz=\bfu_2\).
For the bound, every sign vector \(\bfy=\mathbf{1}_S-\mathbf{1}_{S^c}\) with \(|S|=n/2\) satisfies the relaxed constraints, so \(4\operatorname{cut}(S)=\bfy^\top\bfL\bfy\geq n\lambda_2\).
The eigenvector \(\bfu_2\) is called the Fiedler vector, after Miroslav Fiedler, who studied it in the 1970s, and \(\lambda_2\) is the algebraic connectivity of \(G\). It is not a sign vector, but its entries suggest one: round it by putting vertex \(i\) in \(S\) when \(u_2(i)>0\), or, to keep the split exactly balanced, by putting the \(n/2\) largest entries in \(S\). Think of \(\bfu_2\) as a soft indicator: the sign of \(u_2(i)\) says which side vertex \(i\) belongs to, and its size says how confidently.
This strategy is called relax and round: replace a hard combinatorial constraint by a softer one that linear algebra can handle, solve the relaxed problem with eigenvectors, and convert the answer back. Since the relaxed problem has more candidates, its minimum is always a lower bound for the original one, which tells us how far from optimal any partition we find can be. Relax and round is a standard way to attack hard combinatorial problems; the PCA lower bound for \(k\)-means applies it to \(k\)-means itself.
Why should rounding give a good cut? If \(G\) were disconnected into two pieces \(C_1,C_2\) of equal size, the relaxation would be exact: \(\mathbf{1}_{C_1}-\mathbf{1}_{C_2}\) is a sign vector with roughness \(0\). For a graph that is almost disconnected, the heuristic from Proposition 1 says the Fiedler vector should be almost a sign vector. We make the first half of this precise for \(k\) clusters below.
More than two clusters
The rest of this page was not covered in class. In class, spectral clustering for \(k\) clusters is done in its normalized form in Lecture 16. The sections below are the unnormalized version, with a full relaxation argument for \(k\) clusters, for reference.
One eigenvector splits the graph in two. For \(k\) clusters, we use \(k\) eigenvectors, and the relax-and-round argument carries over.
Encode a partition \(C_1,\ldots,C_k\) of the vertices by the matrix \(\bfH=[\bfh_1,\ldots,\bfh_k]\in\Rbb^{n\times k}\) with columns \(\bfh_\ell=\mathbf{1}_{C_\ell}/\sqrt{|C_\ell|}\). The columns have disjoint supports and unit length, so \(\bfH^\top\bfH=\bfI_k\). By the cut identity, \(\bfh_\ell^\top\bfL\bfh_\ell=\operatorname{cut}(C_\ell)/|C_\ell|\), so \[ \operatorname{tr}(\bfH^\top\bfL\bfH)=\sum_{\ell=1}^k\frac{\operatorname{cut}(C_\ell)}{|C_\ell|}. \] This ratio cut is small when every cluster has few edges leaving it relative to its size. Dividing by \(|C_\ell|\) punishes tiny clusters, a softer version of the balance condition \(|S|=n/2\). Minimizing it over partitions is again a hard combinatorial problem, so we relax: allow any \(\bfY\in\Rbb^{n\times k}\) with orthonormal columns.
Proof. The equality on the left is the Laplacian quadratic form applied to each column of \(\bfY\). The bound is Proposition 2. For the consequence, \(\bfH\) satisfies \(\bfH^\top\bfH=\bfI_k\).
The left-hand side says how to read \(\bfU_k\): placing vertex \(i\) at the row \(\bfy_i\in\Rbb^k\) is the smoothest way to put the vertices in \(\Rbb^k\), in the sense that heavily connected vertices land close together. The constraint \(\bfY^\top\bfY=\bfI_k\) rules out the trivial answer of putting every vertex at the same point.
If \(G\) is connected, \(\bfu_1\) is constant, so the first coordinate shifts every point by the same amount and does not affect the clustering. We keep it anyway, because when \(G\) is disconnected the eigenvalue \(0\) is repeated and an eigensolver may return any orthonormal basis of \(\ker\bfL\).
To round, look at what the rows of \(\bfH\) look like: every vertex of \(C_\ell\) sits at the same point \(\bfe_\ell/\sqrt{|C_\ell|}\). So a partition is an embedding in which each cluster has collapsed to a single point. Rounding the relaxed solution \(\bfU_k\) back to a partition means grouping its rows into \(k\) tight groups, and that is what \(k\)-means does.
Why not round each eigenvector by sign separately, as we did with the Fiedler vector? The relaxed problem does not single out the columns of \(\bfU_k\): for any orthogonal \(\bfQ\in\Rbb^{k\times k}\), the matrix \(\bfU_k\bfQ\) also has orthonormal columns and \(\operatorname{tr}(\bfQ^\top\bfU_k^\top\bfL\bfU_k\bfQ)=\operatorname{tr}(\bfU_k^\top\bfL\bfU_k)\), so it is optimal too. So a single eigenvector need not look like any one cluster indicator. With three clusters, \(\bfu_2\) might be positive on \(C_1\) and \(C_2\) and negative on \(C_3\), and rounding it by sign merges \(C_1\) and \(C_2\). What a rotation does not change is the distance between rows, since row \(i\) of \(\bfU_k\bfQ\) is \(\bfQ^\top\bfy_i\) and \(\|\bfQ^\top(\bfy_i-\bfy_j)\|_2=\|\bfy_i-\bfy_j\|_2\). So the rounding should look at the rows, not the columns, and \(k\)-means on the rows does exactly that.
Why the embedding reveals clusters
In the ideal case the relaxation is exact.
Proof. By Proposition 1, \(\ker\bfL=\operatorname{span}\{\mathbf{1}_{C_1},\ldots,\mathbf{1}_{C_k}\}\) has dimension \(k\), so the eigenvalue \(0\) appears exactly \(k\) times and \(\operatorname{span}\{\bfu_1,\ldots,\bfu_k\}=\ker\bfL\). Both \(\{\bfu_1,\ldots,\bfu_k\}\) and the indicators are bases of this space, so \(\bfU_k=\bfH\bfC\) for the indicator matrix \(\bfH=[\mathbf{1}_{C_1},\ldots,\mathbf{1}_{C_k}]\) and an invertible \(\bfC\in\Rbb^{k\times k}\). Row \(i\) of \(\bfH\bfC\) is row \(\ell\) of \(\bfC\) whenever \(i\in C_\ell\), so each component goes to a single point. The rows of an invertible matrix are distinct, so different components go to different points.
Real graphs are not disconnected, but data with clusters gives a graph that is close to a disconnected one: a few weak edges between pieces. Eigenvalues and eigenspaces move continuously with the matrix, so \(\lambda_2,\ldots,\lambda_k\) stay close to \(0\) and each cluster lands in a small blob near one of \(k\) distinct points (making this quantitative is matrix perturbation theory, which we do not cover). Small, well-separated blobs are exactly what \(k\)-means handles well. The embedding has turned connectivity, which \(k\)-means cannot see, into Euclidean distance, which it can.
Spectral clustering
Putting the pieces together gives the algorithm.
Only the graph construction looks at the original coordinates; everything after it sees only \(\bfW\).
import numpy as np
from scipy.spatial.distance import cdist
from sklearn.cluster import KMeans
from sklearn.datasets import make_circles
def spectral_embedding(W, k):
L = np.diag(W.sum(axis=1)) - W
lam, U = np.linalg.eigh(L) # increasing order
return lam[:k], U[:, :k]
X, y = make_circles(n_samples=400, factor=0.4, noise=0.05, random_state=0)
W = np.exp(-cdist(X, X, "sqeuclidean") / (2 * 0.01)) # epsilon = 0.01
np.fill_diagonal(W, 0)
lam, U = spectral_embedding(W, 2)
labels = KMeans(n_clusters=2, n_init=10, random_state=0).fit_predict(U)With \(\varepsilon=0.01\) we get \(\lambda_2\approx4\times10^{-5}\), a nearly disconnected graph, and the labels match the two rings exactly. \(k\)-means on the raw coordinates, as in the homework, agrees with the rings on only \(51\%\) of points, no better than a coin flip. With \(\varepsilon=0.1\), the kernel is wide enough to connect the rings, \(\lambda_2\) jumps to about \(6.8\), and spectral clustering fails too, agreeing with the rings on \(55\%\) of points. Too small an \(\varepsilon\) fails differently: the graph shatters into many small components, and the smallest eigenvectors find those instead. The real modelling decision is the graph; see also the notes on choosing hyperparameters.
In practice, one uses a sparse nearest-neighbour graph and an iterative eigensolver that only needs products \(\bfL\bfx\), in the spirit of the power method, instead of a dense \(O(n^3)\) eigendecomposition. Most implementations, including scikit-learn’s SpectralClustering, also weight each vertex by its degree. Lecture 16 covers that version.
Summary
The functions with zero roughness are exactly those constant on connected components (Proposition 1). Eigenvectors of symmetric matrices solve trace optimization problems, with the smallest eigenvalues for minimization (Proposition 2) and the largest for maximization (Corollary 1). Writing a balanced cut as a quadratic form on sign vectors and relaxing to real vectors gives the Fiedler vector of \(\bfL\), which we round by sign (Proposition 3). For \(k\) clusters, relaxing the ratio cut gives the \(k\) smallest eigenvectors of \(\bfL\), whose rows are the smoothest embedding of the vertices in \(\Rbb^k\) (Proposition 4), and rounding becomes \(k\)-means. When the graph is disconnected the relaxation is exact, and each component collapses to a single point (Proposition 5). Spectral clustering is \(k\)-means on the spectral embedding (alg. 1).