Hilbert spaces, orthogonality and projections

A Banach space has lengths but no angles. Adding an inner product gives angles and, with them, orthogonality, which is the single most useful structure in analysis. Hilbert spaces are the Banach spaces whose norm comes from an inner product, and they behave much more like Euclidean space than general Banach spaces do. This lesson proves the Cauchy–Schwarz inequality and the parallelogram law, shows that the parallelogram law characterizes inner product norms, and proves the projection theorem: every closed convex set has a unique closest point to any given vector. The projection theorem gives orthogonal complements and orthogonal projections, and it is the engine behind least squares, the Riesz representation theorem of lesson 6 and the existence of weak solutions to partial differential equations in lesson 12.

Inner products

Definition 4.1. An inner product on a vector space \(H\) over \(\mathbb{K}\) is a map \(\langle \cdot, \cdot \rangle : H \times H \to \mathbb{K}\) that is linear in the first variable, conjugate symmetric, \(\langle y, x \rangle = \overline{\langle x, y \rangle}\), and positive definite, \(\langle x, x \rangle > 0\) for \(x \ne 0\). It follows that the inner product is conjugate linear in the second variable. We write \(\|x\| = \langle x, x \rangle^{1/2}\). A Hilbert space is an inner product space that is complete in this norm.

Conventions differ: physicists usually take inner products linear in the second variable. We follow the mathematical convention. The basic examples are \(\mathbb{K}^n\) with \(\langle x, y \rangle = \sum x_i \overline{y_i}\); the sequence space \(\ell^2\) with \(\langle x, y \rangle = \sum_n x_n \overline{y_n}\), which converges absolutely by the Cauchy–Schwarz inequality for sums; and \(L^2(\mu)\) with \(\langle f, g \rangle = \int f \overline{g} \, d\mu\), complete by the Riesz–Fischer theorem. Weighted spaces \(L^2(w \, dx)\) and the Sobolev space \(H^1(a, b)\) with \(\langle f, g \rangle = \int (f \overline{g} + f^{\prime} \overline{g^{\prime}})\) are further examples that appear later.

Theorem 4.2 (Cauchy–Schwarz inequality). In an inner product space, \(|\langle x, y \rangle| \le \|x\| \, \|y\|\), with equality if and only if \(x\) and \(y\) are linearly dependent.

Proof. If \(y = 0\) both sides vanish. Otherwise, for any scalar \(t\), \[0 \le \|x - t y\|^2 = \|x\|^2 - \overline{t} \langle x, y \rangle - t \overline{\langle x, y \rangle} + |t|^2 \|y\|^2.\] Choose \(t = \langle x, y \rangle / \|y\|^2\). The right side becomes \(\|x\|^2 - |\langle x, y \rangle|^2 / \|y\|^2\), which gives the inequality. Equality holds exactly when \(x - ty = 0\) for this \(t\), that is, when \(x\) is a multiple of \(y\). \(\blacksquare\)

Corollary 4.3. The function \(\|x\| = \langle x, x \rangle^{1/2}\) is a norm, and the inner product is continuous on \(H \times H\). Proof: \(\|x + y\|^2 = \|x\|^2 + 2 \operatorname{Re} \langle x, y \rangle + \|y\|^2 \le (\|x\| + \|y\|)^2\) by Cauchy–Schwarz. Continuity follows from \(|\langle x, y \rangle - \langle x^{\prime}, y^{\prime} \rangle| \le \|x - x^{\prime}\| \, \|y\| + \|x^{\prime}\| \, \|y - y^{\prime}\|\). \(\blacksquare\)

The parallelogram law

Expanding \(\|x + y\|^2\) and \(\|x - y\|^2\) and adding gives the parallelogram law \[\|x + y\|^2 + \|x - y\|^2 = 2\|x\|^2 + 2\|y\|^2:\] the sum of the squares of the diagonals of a parallelogram equals the sum of the squares of its four sides. Subtracting instead recovers the inner product from the norm through the polarization identities: in the real case \(\langle x, y \rangle = \frac{1}{4}(\|x + y\|^2 - \|x - y\|^2)\), and in the complex case \(\langle x, y \rangle = \frac{1}{4} \sum_{k=0}^{3} i^k \|x + i^k y\|^2\).

Theorem 4.4 (Jordan and von Neumann, 1935). A norm comes from an inner product if and only if it satisfies the parallelogram law. One direction we have just seen. For the other, one defines the inner product by polarization and checks the axioms; additivity in the first variable is the only nontrivial step and uses the parallelogram law repeatedly, followed by continuity to pass from rational to real scalars. We omit the details. The theorem gives a quick test. In \(\ell^p\) with \(x = e_1\) and \(y = e_2\), \(\|x + y\|_p^2 + \|x - y\|_p^2 = 2 \cdot 2^{2/p}\), while \(2\|x\|_p^2 + 2\|y\|_p^2 = 4\). These agree only for \(p = 2\), so \(\ell^p\) is a Hilbert space only for \(p = 2\). Similarly \(C[0, 1]\) with the supremum norm fails the law for \(x = 1\) and \(y = t\), so it is not a Hilbert space.

Orthogonality

Vectors \(x, y\) are orthogonal, \(x \perp y\), if \(\langle x, y \rangle = 0\). For orthogonal vectors the Pythagorean theorem \(\|x + y\|^2 = \|x\|^2 + \|y\|^2\) holds, and by induction \(\|\sum_{k=1}^{n} x_k\|^2 = \sum_{k=1}^{n} \|x_k\|^2\) for pairwise orthogonal vectors. For a subset \(A \subseteq H\), the orthogonal complement is \(A^\perp = \{x : \langle x, a \rangle = 0 \text{ for all } a \in A\}\). It is always a closed subspace, being the intersection of the kernels of the continuous functionals \(x \mapsto \langle x, a \rangle\), and \(A^\perp = (\overline{\operatorname{span} A})^\perp\) by continuity of the inner product.

The projection theorem

Theorem 4.5 (closest point). Let \(C\) be a nonempty closed convex subset of a Hilbert space \(H\), and let \(x \in H\). There is a unique \(c_0 \in C\) with \(\|x - c_0\| = \operatorname{dist}(x, C) = \inf_{c \in C} \|x - c\|\).

Proof. Let \(d = \operatorname{dist}(x, C)\) and choose \(c_n \in C\) with \(\|x - c_n\| \to d\). Apply the parallelogram law to the vectors \(x - c_n\) and \(x - c_m\): \[\|c_n - c_m\|^2 = 2\|x - c_n\|^2 + 2\|x - c_m\|^2 - 4\Big\|x - \frac{c_n + c_m}{2}\Big\|^2.\] By convexity \(\frac{1}{2}(c_n + c_m) \in C\), so the last norm is at least \(d\), and \[\|c_n - c_m\|^2 \le 2\|x - c_n\|^2 + 2\|x - c_m\|^2 - 4d^2 \to 2d^2 + 2d^2 - 4d^2 = 0.\] Thus \((c_n)\) is Cauchy, and by completeness it converges to some \(c_0\), which lies in \(C\) because \(C\) is closed. Continuity of the norm gives \(\|x - c_0\| = d\). If \(c_0^{\prime}\) is another closest point, the same inequality applied to the sequence \(c_0, c_0^{\prime}, c_0, c_0^{\prime}, \ldots\) gives \(\|c_0 - c_0^{\prime}\|^2 \le 2d^2 + 2d^2 - 4d^2 = 0\). \(\blacksquare\)

Every hypothesis is needed. Without closedness the infimum need not be attained (take an open disc). Without convexity the closest point need not be unique (take a circle and its center). Without completeness the minimizing sequence may have no limit. In Banach spaces that are not Hilbert spaces, closest points can fail to exist even for closed subspaces, and they can be non-unique: in \(\mathbb{R}^2\) with the supremum norm, every point \((t, 1)\) with \(|t| \le 1\) is a closest point of the horizontal axis to \((0, 1)\).

Theorem 4.6 (projection theorem). Let \(M\) be a closed subspace of a Hilbert space \(H\). For each \(x \in H\) the closest point \(Px \in M\) is characterized by \(x - Px \perp M\). Every \(x\) has a unique decomposition \(x = m + n\) with \(m \in M\) and \(n \in M^\perp\), namely \(m = Px\). Thus \(H = M \oplus M^\perp\). The map \(P\) is a bounded linear operator with \(P^2 = P\), \(\langle Px, y \rangle = \langle x, Py \rangle\), \(\|P\| = 1\) unless \(M = \{0\}\), range \(M\) and kernel \(M^\perp\). Moreover \(M^{\perp\perp} = M\).

Proof. Let \(m_0 = Px\) and \(z = x - m_0\). For \(m \in M\) and real \(t\), \(m_0 + t m \in M\), so the function \(\phi(t) = \|z - tm\|^2 = \|z\|^2 - 2t \operatorname{Re} \langle z, m \rangle + t^2 \|m\|^2\) has a minimum at \(t = 0\). Hence \(\operatorname{Re} \langle z, m \rangle = 0\); replacing \(m\) by \(im\) gives \(\operatorname{Im} \langle z, m \rangle = 0\). So \(z \perp M\). Conversely if \(m_0 \in M\) and \(x - m_0 \perp M\), then for all \(m \in M\), Pythagoras gives \(\|x - m\|^2 = \|x - m_0\|^2 + \|m_0 - m\|^2 \ge \|x - m_0\|^2\), so \(m_0\) is the closest point. The decomposition exists with \(m = Px\), and it is unique because \(M \cap M^\perp = \{0\}\): a vector orthogonal to itself is zero. Linearity of \(P\) follows from uniqueness: \(\alpha x + \beta y = (\alpha Px + \beta Py) + (\text{an element of } M^\perp)\). Pythagoras gives \(\|x\|^2 = \|Px\|^2 + \|x - Px\|^2 \ge \|Px\|^2\), so \(\|P\| \le 1\), with equality on \(M\). Finally \(\langle Px, y \rangle = \langle Px, Py \rangle = \langle x, Py \rangle\), and if \(x \in M^{\perp\perp}\) then \(x - Px \in M^\perp \cap M^{\perp\perp} = \{0\}\), so \(x \in M\). \(\blacksquare\)

Corollary 4.7. A subspace \(M\) of a Hilbert space is dense if and only if \(M^\perp = \{0\}\). Proof: \(\overline{M}^\perp = M^\perp\), and \(\overline{M} = (\overline{M})^{\perp\perp}\), which is \(H\) exactly when \(\overline{M}^\perp = \{0\}\). \(\blacksquare\) This is the standard way to prove density in Hilbert spaces: show that the only vector orthogonal to everything in \(M\) is zero. Lesson 5 uses it to prove that the trigonometric system is complete.

For the projection onto a finite-dimensional subspace with a basis \(v_1, \ldots, v_n\), write \(Px = \sum_j c_j v_j\). The condition \(x - Px \perp v_i\) for each \(i\) gives the normal equations \[\sum_{j=1}^{n} \langle v_j, v_i \rangle c_j = \langle x, v_i \rangle, \qquad i = 1, \ldots, n.\] The matrix \(G_{ij} = \langle v_j, v_i \rangle\) is the Gram matrix; it is invertible because the \(v_j\) are independent. If the basis is orthonormal, \(G = I\) and \(Px = \sum_j \langle x, v_j \rangle v_j\). Least squares fitting in statistics, the best polynomial approximation in \(L^2\) and the Galerkin method for differential equations are all instances of this formula.

# Best L^2[0,1] approximation of exp(x) by polynomials of degree < n via the normal equations.
from fractions import Fraction
import math

def solve(G, b):
    """Gaussian elimination with exact fractions."""
    n = len(b); M = [row[:] + [b[i]] for i, row in enumerate(G)]
    for c in range(n):
        p = next(r for r in range(c, n) if M[r][c] != 0); M[c], M[p] = M[p], M[c]
        for r in range(n):
            if r != c and M[r][c] != 0:
                f = M[r][c] / M[c][c]
                M[r] = [a - f * b for a, b in zip(M[r], M[c])]
    return [M[i][n] / M[i][i] for i in range(n)]

e = Fraction(math.e).limit_denominator(10 ** 15)
def moment(k):
    """integral_0^1 x^k e^x dx by the recursion I_k = e - k I_(k-1), using exact arithmetic."""
    I = e - 1
    for j in range(1, k + 1):
        I = e - j * I
    return I

norm2 = (e * e - 1) / 2                      # integral of e^(2x)
for n in (1, 2, 3, 4, 5):
    G = [[Fraction(1, i + j + 1) for j in range(n)] for i in range(n)]   # Hilbert matrix
    b = [moment(i) for i in range(n)]
    c = solve(G, b)
    err2 = norm2 - sum(ci * bi for ci, bi in zip(c, b))                    # Pythagoras
    print(f"degree < {n}: coefficients {[round(float(t), 4) for t in c]}, L2 error = {math.sqrt(float(err2)):.2e}")

degree < 1: coefficients [1.7183], L2 error = 4.92e-01
degree < 2: coefficients [0.8731, 1.6903], L2 error = 6.28e-02
degree < 3: coefficients [1.013, 0.8511, 0.8392], L2 error = 5.28e-03
degree < 4: coefficients [0.9991, 1.0183, 0.4212, 0.2786], L2 error = 3.31e-04
degree < 5: coefficients [1.0001, 0.9984, 0.5106, 0.1397, 0.0695], L2 error = 1.66e-05

The error was computed by Pythagoras: \(\|x - Px\|^2 = \|x\|^2 - \|Px\|^2 = \|x\|^2 - \sum_j c_j \langle v_j, x \rangle\). The Gram matrix of the monomials \(1, x, \ldots, x^{n-1}\) in \(L^2[0, 1]\) has entries \(\frac{1}{i + j + 1}\), the notorious Hilbert matrix, studied by Hilbert in 1894. It becomes extremely ill conditioned as \(n\) grows, which is why the code uses exact fractions. Lesson 5 avoids the problem altogether by orthonormalizing the monomials first, which produces the Legendre polynomials.

Two consequences

First, Riesz's lemma holds with \(\theta = 1\) in Hilbert spaces: if \(M\) is a closed proper subspace, pick \(x \notin M\), and \(z = (x - Px) / \|x - Px\|\) is a unit vector in \(M^\perp\), so \(\|z - m\|^2 = 1 + \|m\|^2 \ge 1\) for all \(m \in M\). Second, every closed subspace of a Hilbert space is complemented: \(M^\perp\) is a closed subspace with \(H = M \oplus M^\perp\) and bounded projections onto each. In Banach spaces this fails badly. Phillips proved in 1940 that \(c_0\) has no bounded projection from \(\ell^\infty\) onto it, and Lindenstrauss and Tzafriri proved in 1971 that a Banach space in which every closed subspace is complemented must be isomorphic to a Hilbert space.