Completing the Square in Several Variables

I often find myself staring at quadratic multivariable functions. A common symptom for the helpless computational mathematicians working with nonlinear methods. Completing the square here often helps but, alas, I haven’t memorized it yet and I’m tired of digging up my old notes for it.

Basic Version

Theorem 1 Let \(A \in \mathbb R^{n\times n}\) be symmetric and invertible, and let \(b,z\in \mathbb R^n\) and \(c\in \mathbb R\). Then \[ \frac{1}{2}z^T A z + b^T z + c = \frac{1}{2}(z + A^{-1}b)^T A (z + A^{-1}b) + c - \frac{1}{2}b^T A^{-1}b. \tag{1}\]

Proof. Start with the proposed squared term and expand it: \[ \begin{aligned} \frac{1}{2}(z + A^{-1}b)^T A (z + A^{-1}b) &= \frac{1}{2} \left( z^T A z + z^T A A^{-1}b + (A^{-1}b)^T A z + (A^{-1}b)^T A A^{-1}b \right) \\ &= \frac{1}{2}z^T A z + \frac{1}{2}z^T b + \frac{1}{2}b^T z + \frac{1}{2}b^T A^{-1}b. \end{aligned} \] Here we used \(A^T=A\) and \((A^{-1})^T=A^{-1}\), which follow from symmetry and invertibility of \(A\). Since \(z^Tb=b^Tz\) is a scalar, \[ \frac{1}{2}(z + A^{-1}b)^T A (z + A^{-1}b) = \frac{1}{2}z^T A z + b^T z + \frac{1}{2}b^T A^{-1}b. \] Rearranging gives \[ \frac{1}{2}z^T A z + b^T z + c = \frac{1}{2}(z + A^{-1}b)^T A (z + A^{-1}b) + c - \frac{1}{2}b^T A^{-1}b, \] which proves the identity.

Symmetric Positive Definite Corollary

If \(A\) is symmetric positive definite, then \(\|x\|_A^2 := x^T A x\) is a norm. Then, (Equation 1) can be written as \[ \frac{1}{2}z^T A z + b^T z + c = \frac{1}{2}\|z + A^{-1}b\|_A^2 + c - \frac{1}{2}b^T A^{-1}b. \] In this case, the stationary point \(z_* = -A^{-1}b\) is also the global minimizer, hence it’s often useful to view the identity centered around it: \[ \frac{1}{2}z^T A z + b^T z + c = \frac{1}{2}\|z-z_*\|_A^2 + c - \frac{1}{2}b^T A^{-1}b. \]