Normalized cuts
Math 123: Mathematical Aspects of Data Analysis – Fall 2026
Where we are
Before the midterm we split a graph in two by relax and round. We wrote a balanced cut as a quadratic form, \(\bfy^\top\bfL\bfy=4\operatorname{cut}(S)\) for the sign vector \(\bfy=\mathbf{1}_S-\mathbf{1}_{S^c}\), dropped the requirement that \(\bfy\) take only the values \(\pm1\), and rounded the Fiedler vector by sign.
Today we finish spectral clustering with one change: measure the size of each side by its total degree instead of its number of vertices. The same relax-and-round argument, now with a normalized Laplacian, gives the algorithm used in practice, for any number of clusters \(k\). This uses only the tools of Lecture 14, and follows Sections 5.4–5.5 of the textbook. Throughout, \(G\) is a weighted graph with no isolated vertices, so \(\deg(i)>0\) for every \(i\) and \(\bfD\) is invertible, and \(\operatorname{vol}(S)=\sum_{i\in S}\deg(i)\).
The normalized cut
Counting vertices treats every vertex the same, but in a similarity graph they are patently not! A point deep inside a dense region has large degree, while an outlier has degree near zero. The volume \(\operatorname{vol}(S)\) counts the total similarity attached to \(S\), and \(\operatorname{cut}(S)/\operatorname{vol}(S)\) is the fraction of that similarity which leaves \(S\).
Like the balanced cut, the normalized cut is a quadratic form in \(\bfL\) on two-valued vectors. The only change is in the encoding: instead of the values \(\pm1\), use values \(a\) on \(S\) and \(b\) on \(S^c\), chosen so that the two sides balance by volume, \[ a\operatorname{vol}(S)+b\operatorname{vol}(S^c)=0, \qquad a^2\operatorname{vol}(S)+b^2\operatorname{vol}(S^c)=1. \] In vector form these say \(\bfy^\top\bfD\mathbf{1}=0\) and \(\bfy^\top\bfD\bfy=1\), since \(\bfy^\top\bfD\mathbf{1}=\sum_i\deg(i)y_i\) and \(\bfy^\top\bfD\bfy=\sum_i\deg(i)y_i^2\).
Proof. Write \(s=\operatorname{vol}(S)\) and \(s^c=\operatorname{vol}(S^c)\), so \(s+s^c=\operatorname{vol}(V)\). Then \(as=\sqrt{ss^c/\operatorname{vol}(V)}=-bs^c\), which gives the first constraint, and \(a^2s+b^2s^c=(s^c+s)/\operatorname{vol}(V)=1\), which gives the second. By the Laplacian quadratic form, only edges crossing the cut contribute, each with \((a-b)^2\), so \(\bfy^\top\bfL\bfy=(a-b)^2\operatorname{cut}(S)\). Finally, \(ab=-1/\operatorname{vol}(V)\), so \[ (a-b)^2=a^2+b^2-2ab=\frac1{\operatorname{vol}(V)}\Big(\frac{s^c}{s}+\frac{s}{s^c}+2\Big)=\frac{(s+s^c)^2}{ss^c\operatorname{vol}(V)}=\frac1s+\frac1{s^c}. \]
Side by side with the balanced cut:
| Balanced cut | Normalized cut | |
|---|---|---|
| Values of \(\bfy\) | \(\pm1\) | \(a\) on \(S\), \(b\) on \(S^c\) |
| Balance | \(\bfy^\top\mathbf{1}=0\) | \(\bfy^\top\bfD\mathbf{1}=0\) |
| Scale | \(\bfy^\top\bfy=n\) | \(\bfy^\top\bfD\bfy=1\) |
| Objective | \(\bfy^\top\bfL\bfy=4\operatorname{cut}(S)\) | \(\bfy^\top\bfL\bfy=\operatorname{Ncut}(S)\) |
Relaxing the normalized cut
Minimizing \(\operatorname{Ncut}(S)\) over all sets is again a hard combinatorial problem. As before, relax: keep the two constraints, but allow any \(\bfy\in\Rbb^n\). The weights \(\bfD\) in the constraints are handled by the change of variables \(\bfz=\bfD^{1/2}\bfy\), which turns \(\bfy^\top\bfL\bfy\) into \(\bfz^\top\bfL_{\mathrm{sym}}\bfz\) for the normalized Laplacian \[ \bfL_{\mathrm{sym}}=\bfD^{-1/2}\bfL\bfD^{-1/2}=\bfI-\bfD^{-1/2}\bfW\bfD^{-1/2}. \] It is symmetric, so it has an orthonormal basis of eigenvectors, and it is positive semidefinite, since \(\bfz^\top\bfL_{\mathrm{sym}}\bfz=\bfy^\top\bfL\bfy\geq0\).
Proof. First, \(\bfL_{\mathrm{sym}}\bfD^{1/2}\mathbf{1}=\bfD^{-1/2}\bfL\mathbf{1}=\mathbf{0}\) and \(\|\bfD^{1/2}\mathbf{1}\|_2^2=\sum_i\deg(i)=\operatorname{vol}(V)\), so \(\bfv_1\) is a unit eigenvector for the eigenvalue \(0\), which is the smallest since \(\bfL_{\mathrm{sym}}\) is positive semidefinite.
With \(\bfz=\bfD^{1/2}\bfy\), the constraints become \(\|\bfz\|_2^2=\bfy^\top\bfD\bfy=1\) and \(\bfz^\top\bfD^{1/2}\mathbf{1}=\bfy^\top\bfD\mathbf{1}=0\), that is, \(\bfz\perp\bfv_1\). The objective becomes \(\bfy^\top\bfL\bfy=\bfz^\top\bfL_{\mathrm{sym}}\bfz\). By Rayleigh–Ritz for a single vector, the minimum is \(\lambda_2\), attained at \(\bfz=\bfv_2\), that is, \(\bfy=\bfD^{-1/2}\bfv_2\).
For the consequence, the two-valued \(\bfy\) of Proposition 1 satisfies the relaxed constraints.
In general we write \(\bfphi_\ell=\bfD^{-1/2}\bfv_\ell\) for every \(\ell\); these pulled-back eigenvectors are the ones we cluster with. Note that \(\bfphi_1=\mathbf{1}/\sqrt{\operatorname{vol}(V)}\) is constant.
To round, threshold: put vertex \(i\) in \(S\) when \(\phi_2(i)\leq\tau\). If \(\bfphi_2\) were exactly two-valued, taking \(a>0\) and \(b<0\) as in Proposition 1, then \(\tau=0\), rounding by sign as for the Fiedler vector, would separate the two sides exactly. But \(\bfphi_2\) only solves the relaxed problem, so its entries spread out around \(a\) and \(b\), and the gap between the two groups need not sit at \(0\). Unequal volumes make this worse: since \(a/|b|=\operatorname{vol}(S^c)/\operatorname{vol}(S)\), the larger side takes the value closer to \(0\), and its spread can straddle \(0\). One can try every threshold and keep the best one, or simply run \(k\)-means with \(k=2\) on the numbers \(\phi_2(1),\ldots,\phi_2(n)\).
Remark (Cheeger’s inequality). Relax and round usually gives only a lower bound on the best cut, with no guarantee for the rounded answer. Here there is one. Let \(h_G=\min_S\operatorname{cut}(S)/\min\{\operatorname{vol}(S),\operatorname{vol}(S^c)\}\) be the Cheeger constant of a connected graph. Then \[ \frac{\lambda_2}2\leq h_G\leq\sqrt{2\lambda_2}, \] and the upper bound is achieved by thresholding \(\bfphi_2\) at some \(\tau\). The lower bound follows from Proposition 2 and \(\operatorname{Ncut}(S)\leq2\operatorname{cut}(S)/\min\{\operatorname{vol}(S),\operatorname{vol}(S^c)\}\); the upper bound is a rounding guarantee. See Section 5.5 of the textbook.
More than two clusters
For \(k\) clusters, use the first \(k\) eigenvectors \(\bfphi_1,\ldots,\bfphi_k\) instead of just \(\bfphi_2\), and read them row by row: each vertex gets one coordinate from each eigenvector.
On a connected graph the first coordinate \(\phi_1(i)=1/\sqrt{\operatorname{vol}(V)}\) is the same for every vertex, so it shifts all points equally and does not change any distance between them. The ideal case, a graph that falls apart into \(k\) pieces, shows why the embedding reveals clusters.
Proof. Since \(\bfD^{-1/2}\) is invertible, \(\bfL_{\mathrm{sym}}\bfv=\bfD^{-1/2}\bfL\bfD^{-1/2}\bfv\) is zero exactly when \(\bfL(\bfD^{-1/2}\bfv)=\mathbf{0}\). So \(\bfv\mapsto\bfD^{-1/2}\bfv\) maps \(\ker\bfL_{\mathrm{sym}}\) one-to-one onto \(\ker\bfL\), which is spanned by the component indicators and has dimension \(k\). Hence the eigenvalue \(0\) of \(\bfL_{\mathrm{sym}}\) appears exactly \(k\) times, and \(\bfphi_1,\ldots,\bfphi_k\) is a basis of \(\ker\bfL\). So \([\bfphi_1,\ldots,\bfphi_k]=[\mathbf{1}_{C_1},\ldots,\mathbf{1}_{C_k}]\bfC\) for an invertible \(\bfC\in\Rbb^{k\times k}\), and row \(i\) is row \(\ell\) of \(\bfC\) whenever \(i\in C_\ell\). The rows of an invertible matrix are distinct, so different components go to different points.
Notice that we look at the rows, not the individual eigenvectors. The eigenvalue \(0\) is repeated, so an eigensolver may return any orthonormal basis of its eigenspace, and a single \(\bfphi_\ell\) need not look like any one indicator: with three components, \(\bfphi_2\) might be positive on two of them. The rows, however, collapse to \(k\) points whatever basis we get. For a graph that is nearly disconnected, the rows land in \(k\) small blobs instead of \(k\) points, and grouping them is a job for \(k\)-means.
For \(k=2\) this is the thresholding of \(\bfphi_2\) from before, since \(\phi_1\) is constant on a connected graph. This is the Shi–Malik algorithm (Shi and Malik 2000), and it is essentially what scikit-learn’s SpectralClustering does. There is also a \(k\)-cluster version of Proposition 2, which needs Rayleigh–Ritz for several vectors at once; it falls out of Laplacian eigenmaps in Lecture 17.
import numpy as np
from scipy.spatial.distance import cdist
from sklearn.cluster import KMeans
from sklearn.datasets import make_circles
def normalized_spectral_embedding(W, k):
d = W.sum(axis=1)
L_sym = np.eye(len(d)) - W / np.sqrt(np.outer(d, d))
lam, V = np.linalg.eigh(L_sym) # increasing order
return lam[:k], V[:, :k] / np.sqrt(d)[:, None] # phi_l = D^{-1/2} v_l
X, y = make_circles(n_samples=400, factor=0.4, noise=0.05, random_state=0)
W = np.exp(-cdist(X, X, "sqeuclidean") / (2 * 0.01)) # epsilon = 0.01
np.fill_diagonal(W, 0)
lam, Phi = normalized_spectral_embedding(W, 2)
labels = KMeans(n_clusters=2, n_init=10, random_state=0).fit_predict(Phi)That completes normalized spectral clustering. Next time we keep the coordinates \(\phi_2(i),\ldots,\phi_{k}(i)\) instead of rounding them, which gives our first method for nonlinear dimension reduction.