Math 123: Mathematical Aspects of Data Analysis – Fall 2026

Published

September 8, 2026

The Basics of CQC Data

Data as Vectors

Definition 1 (Observation and feature vector) An observation is one object being measured. A feature is one numerical measurement of that object.

If one observation has \(d\) real-valued features, its feature vector is \[ \bfx = \begin{pmatrix} x_1 \\ x_2 \\ \vdots \\ x_d \end{pmatrix} \in \Rbb^d. \]

The number \(d\) is the ambient dimension of the data representation.

Example 1 (Tabular observation) One student described by \[ \text{attendance},\quad \text{homework average},\quad \text{exam average},\quad \text{hours studied} \] is represented by one vector \(\bfx \in \Rbb^4\).

Example 2 (Image as a vector) A grayscale \(p \times q\) image is an array of pixel intensities. After choosing an ordering of the pixels, the same image is a vector in \(\Rbb^{pq}\).

An RGB image with three color channels is a vector in \(\Rbb^{3pq}\).

Example 3 (Document as a vector) Fix a vocabulary of \(d\) words. A document may be represented by \[ \bfx = (x_1,\ldots,x_d) \in \Rbb^d, \] where \(x_j\) records the count, frequency, or weighted frequency of word \(j\).

Data as Matrices

Definition 2 (Data matrix) A dataset with \(n\) observations and \(d\) features is stored as a matrix \[ \bfX = \begin{pmatrix} - & \bfx_1^\top & - \\ - & \bfx_2^\top & - \\ & \vdots & \\ - & \bfx_n^\top & - \end{pmatrix} \in \Rbb^{n \times d}. \]

Row \(i\) is observation \(\bfx_i^\top\). Column \(j\) is feature \(j\) across all observations.

TipRemark (Rows versus columns)

Remark (Rows versus columns). Two conventions:

  • \(\bfX \in \Rbb^{n \times d}\): rows are observations, columns are features.
  • \(\bfY \in \Rbb^{d \times n}\): columns are observations, rows are features.

These conventions are related by transposition: \(\bfY = \bfX^\top\).

Definition 3 (Point cloud) The rows of \(\bfX \in \Rbb^{n \times d}\) define a finite point cloud \[ \mathcal{X} = \{\bfx_1,\ldots,\bfx_n\} \subseteq \Rbb^d. \]

Data analysis tasks become geometric tasks on \(\mathcal{X}\):

  • nearest-neighbor search,
  • clustering,
  • classification,
  • regression,
  • denoising,
  • dimension reduction,
  • outlier detection.

Big-Data is Messed Up

Definition 4 (Euclidean balls and cubes) For \(R>0\), the \(d\)-dimensional Euclidean ball of radius \(R\), centered at the origin, is \[ B_2^d(R) = \left\{\bfx \in \Rbb^d : \|\bfx\|_2 \le R\right\}. \]

The \(d\)-dimensional cube of side length \(R\), centered at the origin, is \[ C^d(R) = \left[-\frac{R}{2},\frac{R}{2}\right]^d. \]

Proposition 1 (Volumes of Euclidean balls and cubes) For \(R>0\), \[ \operatorname{vol}\!\left(B_2^d(R)\right) = \frac{\pi^{d/2}}{\Gamma(d/2+1)}R^d, \qquad \operatorname{vol}\!\left(C^d(R)\right) = R^d. \]

For fixed \(R\), Stirling’s approximation gives \[ \operatorname{vol}\!\left(B_2^d(R)\right) \sim \frac{1}{\sqrt{\pi d}} \left(\frac{2\pi e}{d}\right)^{d/2}R^d \qquad \text{as } d \to \infty. \] In particular, the volume of the unit ball \(B_2^d(1)\) vanishes as the dimension grows.

A unit disk containing a centered square of side length one; the square's corner is at distance square root of two over two from the origin.
(a) The unit disk and centered unit square in two dimensions. Source: Bandeira et al. (n.d.), Section 2.1.1.
A unit ball and centered unit cube in high dimension, whose corners are at distance square root of d over two from the origin.
(b) A schematic comparison of the unit ball and centered unit cube in high dimension. Source: Bandeira et al. (n.d.), Section 2.1.1.
Figure 1: The Euclidean unit ball and centered unit cube in low and high dimensions.

Proposition 2 (Volume near the boundary) Let \(0 < \epsilon < 1\). Removing the outermost layer of thickness \(\epsilon\) from the unit ball leaves \(B_2^d(1-\epsilon)\), whose volume relative to the unit ball is \[ \frac{\operatorname{vol}\!\left(B_2^d(1-\epsilon)\right)} {\operatorname{vol}\!\left(B_2^d(1)\right)} = (1-\epsilon)^d. \] Consequently, for any fixed \(\epsilon>0\), this fraction tends to zero as \(d \to \infty\): almost all of the unit ball’s volume lies in its outermost \(\epsilon\) layer.

Descent into Probability

Suppose we have a bunch of data, from the same ‘distribution’ \[ X_1,\ldots,X_n \] with a common mean \(\mu\). The average reading is \[ \overline{X} = \frac1n\sum_{i=1}^n X_i. \]

Our first probability question is the one that will guide both lectures:

Why should \(\overline{X}\) be close to \(\mu\), and how unlikely is it that it is not?

Probability enters data science whenever data are sampled, measurements are noisy, or an algorithm uses randomness. We will build only the vocabulary and tools needed to make statements about those situations.

TipRemark (Probability bootcamp I)

Remark (Probability bootcamp I). By the end of this lecture, we want to be able to:

  • identify random variables, distributions, events, and independent trials;
  • compute simple expectations and use linearity of expectation;
  • explain why averaging independent noise reduces variance;
  • turn expectation or variance information into a tail bound using Markov’s or Chebyshev’s inequality.

Random Variables, Distributions, and Events

Definition 5 (Random variable) A random variable is a numerical quantity whose value is determined by the outcome of a random experiment.

We write a random variable with a capital letter, such as \(X\), and a possible value with a lowercase letter, such as \(x\).

Definition 6 (Distribution and event) The distribution of \(X\) describes how probability is assigned to its possible values. For example, for a discrete random variable (a random variable taking discrete values), we may consider the probability that \(X\) takes on a random event. That’s typically written in the notation \[ \Pbb(X=x) \] for every possible value \(x\).

An event is a yes-or-no statement about the random outcome. For example, \[ \{X\ge 5\} \] is an event, and \(\Pbb(X\ge 5)\) is the probability that it occurs.

Example 4 (Three recurring examples)  

  1. A Bernoulli random variable with parameter \(p\) has distribution \[ X= \begin{cases} 1, & \text{with probability }p,\\ 0, & \text{with probability }1-p. \end{cases} \] For example, you may consider a classifier determining whether an email is spam or not.

  2. A Rademacher random variable is a random sign: \[ \Pbb(X=1)=\Pbb(X=-1)=\frac12. \] Seemingly arbitrary, but random signs appear in randomized linear algebra.

  3. A standard Gaussian random variable, written \(X\sim N(0,1)\), is a continuous random variable with a symmetric bell-shaped distribution centered at \(0\). For continuous variables, probabilities are assigned to intervals by integrating over its associated density function. The standard Gaussian density is \[ \varphi(x) = \frac{1}{\sqrt{2\pi}}e^{-x^2/2}, \qquad x\in\Rbb. \] Therefore, if \(a\leq b\), then \[ \Pbb(a\leq X\leq b) = \int_a^b \varphi(x)\,dx = \Phi(b)-\Phi(a), \] where \(\Phi(t)=\int_{-\infty}^t \varphi(x)\,dx\) is the standard Gaussian cumulative distribution function.

Definition 7 (Independent and identically distributed) Random variables are independent when learning the value of one gives no information about the others. For two events \(A\) and \(B\), independence means \[ \Pbb(A\cap B)=\Pbb(A)\Pbb(B). \]

The abbreviation i.i.d. means independent and identically distributed: the variables are independent and all have the same distribution.

Expectation

Definition 8 (Expectation) The expectation or mean of a discrete random variable is its probability-weighted average: \[ \Ebb[X] = \sum_x x\,\Pbb(X=x). \]

For a continuous random variable, the analogous definition uses an integral. If \(X\) has density \(f\), then \[ \Ebb[X] = \int_{-\infty}^{\infty}x f(x)\,dx. \] Note: Expectation describes the center of a distribution; it need not be a possible value of the random variable.

Example 5 (Bernoulli, Rademacher, and a die) If \(X\) is Bernoulli with parameter \(p\), then \[ \Ebb[X]=1\cdot p+0\cdot(1-p)=p. \]

If \(R\) is Rademacher, then \[ \Ebb[R]=1\cdot\frac12+(-1)\cdot\frac12=0. \]

If \(D\) is a fair six-sided die, then \[ \Ebb[D] = \frac{1+2+3+4+5+6}{6} = 3.5. \]

Proposition 3 (Linearity of expectation) For random variables \(X_1,\ldots,X_n\) and constants \(a_1,\ldots,a_n\), \[ \Ebb\!\left[\sum_{i=1}^n a_iX_i\right] = \sum_{i=1}^n a_i\Ebb[X_i], \] whenever the expectations exist.

Linearity of expectation does not require independence.

Example 6 (Expected number of successes) Let \(X_i\) equal \(1\) when trial \(i\) succeeds and \(0\) otherwise, and suppose each trial succeeds with probability \(p\). The number of successes is \[ S=X_1+\cdots+X_n. \] Whether or not the trials are independent, \[ \Ebb[S] = \sum_{i=1}^n\Ebb[X_i] = np. \]

Returning to repeated measurements with \(\Ebb[X_i]=\mu\), linearity gives \[ \Ebb[\overline{X}] = \frac1n\sum_{i=1}^n\Ebb[X_i] = \mu. \] Thus the average is centered at the right place.

Variance, Covariance, and Averaging

Definition 9 (Variance and standard deviation) The variance of \(X\) is the average squared distance from its mean: \[ \Var(X) = \Ebb[(X-\Ebb[X])^2]. \] Its standard deviation is \[ \sigma(X)=\sqrt{\Var(X)}. \]

The useful computational identity \[ \Var(X)=\Ebb[X^2]-\Ebb[X]^2 \] follows by expanding the square and using linearity of expectation.

For constants \(a,b\in\Rbb\), \[ \Var(aX+b)=a^2\Var(X). \] Adding a constant moves the center without changing the spread; multiplying by \(a\) scales squared deviations by \(a^2\).

Example 7 (Variance of a Bernoulli variable) If \(X\) is Bernoulli with parameter \(p\), then \(X^2=X\). Therefore \[ \Var(X) = \Ebb[X^2]-\Ebb[X]^2 = p-p^2 = p(1-p). \] This is largest when \(p=1/2\) and small when the outcome is nearly certain.

Definition 10 (Covariance) The covariance of \(X\) and \(Y\) is \[ \Cov(X,Y) = \Ebb[(X-\Ebb[X])(Y-\Ebb[Y])]. \] Positive covariance means that \(X\) and \(Y\) tend to be above their means together, while negative covariance means that one tends to be high when the other is low.

For two random variables, \[ \Var(X+Y) = \Var(X)+\Var(Y)+2\Cov(X,Y). \] Independent random variables have covariance \(0\), so their variances add. Note: The converse is not generally true! Covariance \(0\) does not by itself imply independence.

Covariance will reappear later as a matrix that describes the directions in which a point cloud varies. For now, its essential role is to show exactly where independence enters the averaging calculation.

Proposition 4 (Averaging away independent noise) Suppose \(X_1,\ldots,X_n\) are independent, each with mean \(\mu\) and variance \(\sigma^2\). Then \[ \Var(\overline{X}) = \Var\!\left(\frac1n\sum_{i=1}^nX_i\right) = \frac1{n^2}\sum_{i=1}^n\Var(X_i) = \frac{\sigma^2}{n}. \] Consequently, \[ \sigma(\overline{X})=\frac{\sigma}{\sqrt n}. \]

Averaging \(n\) independent measurements reduces the variance by a factor of \(n\) and the standard deviation by a factor of \(\sqrt n\).

Tail Probabilities

A tail probability measures the chance of an unusually large deviation, such as \[ \Pbb(|X-\Ebb[X]|\ge t). \] The event inside the probability is the bad event we want to control. Often we do not need its exact probability; an upper bound is enough.

Theorem 1 (Markov’s inequality) If \(X\) is non-negative and \(t>0\), then \[ \Pbb(X\ge t) \le \frac{\Ebb[X]}{t}. \]

Proof (idea). On the event \(\{X\ge t\}\), the variable \(X\) contributes at least \(t\). Therefore \[ \Ebb[X] \ge t\Pbb(X\ge t). \] Divide by \(t\).

Markov’s inequality is easy and informative, but often loose. Often, we can do a little better:

Corollary 1 (Chebyshev’s inequality) If \(X\) has mean \(\mu\) and variance \(\sigma^2\), then for every \(t>0\), \[ \Pbb(|X-\mu|\ge t) \le \frac{\sigma^2}{t^2}. \]

Proof (from Markov). Apply Markov’s inequality to the non-negative random variable \((X-\mu)^2\): \[ \begin{aligned} \Pbb(|X-\mu|\ge t) &= \Pbb((X-\mu)^2\ge t^2)\\ &\le \frac{\Ebb[(X-\mu)^2]}{t^2} = \frac{\sigma^2}{t^2}. \end{aligned} \]

Example 8 (Chebyshev for a sample mean) Under the assumptions of Proposition 4, \[ \Pbb(|\overline{X}-\mu|\ge \epsilon) \le \frac{\sigma^2}{n\epsilon^2}. \]

For fixed \(\epsilon>0\), the right-hand side tends to \(0\) as \(n\) grows. This is the basic reason that averages of independent measurements settle down.

Example 9 (A sample proportion) Let \(X_i\) indicate whether trial \(i\) succeeds, assume the trials are independent with success probability \(p\), and write \[ \widehat p=\frac1n\sum_{i=1}^nX_i. \] Because \(\Var(X_i)=p(1-p)\le 1/4\), \[ \Pbb(|\widehat p-p|\ge\epsilon) \le \frac{1}{4n\epsilon^2}. \]

References

Bandeira, Afonso, Amit Singer, and Thomas Strohmer. n.d. Mathematics of Data Science. https://people.math.ethz.ch/~abandeira/BandeiraSingerStrohmer-MDS-draft.pdf.