Orthonormal bases and Fourier series

In \(\mathbb{R}^n\) an orthonormal basis lets us read off coordinates by inner products: \(x = \sum_i \langle x, e_i \rangle e_i\). In a Hilbert space the same formula holds with an infinite sum, provided the orthonormal set is complete in the right sense. This lesson proves Bessel's inequality and Parseval's identity, characterizes orthonormal bases, shows that every separable Hilbert space is a copy of \(\ell^2\), constructs the Legendre polynomials by the Gram–Schmidt process, and proves that the trigonometric system is an orthonormal basis of \(L^2(-\pi, \pi)\). Fourier series, which motivated so much of analysis from Fourier's work on heat in 1807 onward, thus become a special case of a general and simple theorem.

Orthonormal sets and Bessel's inequality

A family \((e_\alpha)_{\alpha \in A}\) in a Hilbert space \(H\) is orthonormal if \(\langle e_\alpha, e_\beta \rangle\) is \(1\) for \(\alpha = \beta\) and \(0\) otherwise. For a finite orthonormal set \(e_1, \ldots, e_n\), lesson 4 showed that the orthogonal projection onto its span is \(P_n x = \sum_{k=1}^{n} \langle x, e_k \rangle e_k\), and Pythagoras gives \(\|x\|^2 = \|P_n x\|^2 + \|x - P_n x\|^2 \ge \sum_{k=1}^{n} |\langle x, e_k \rangle|^2\). The numbers \(\langle x, e_k \rangle\) are the Fourier coefficients of \(x\).

Theorem 5.1 (Bessel's inequality). For an orthonormal sequence \((e_k)\) and any \(x \in H\), \[\sum_{k=1}^{\infty} |\langle x, e_k \rangle|^2 \le \|x\|^2.\] Proof: the partial sums are bounded by \(\|x\|^2\), as just shown, and the terms are nonnegative. \(\blacksquare\) In particular \(\langle x, e_k \rangle \to 0\) for every \(x\). For an uncountable orthonormal family, Bessel's inequality applied to finite subfamilies shows that at most \(n^2 \|x\|^2\) indices have \(|\langle x, e_\alpha \rangle| \ge \frac{1}{n}\), so only countably many coefficients of a given \(x\) are nonzero.

Theorem 5.2. Let \((e_k)\) be an orthonormal sequence and \((c_k)\) scalars. The series \(\sum_k c_k e_k\) converges in \(H\) if and only if \(\sum_k |c_k|^2 < \infty\), and then \(\|\sum_k c_k e_k\|^2 = \sum_k |c_k|^2\) and \(\langle \sum_k c_k e_k, e_j \rangle = c_j\).

Proof. For \(m > n\), Pythagoras gives \(\|\sum_{k=n+1}^{m} c_k e_k\|^2 = \sum_{k=n+1}^{m} |c_k|^2\). So the partial sums are Cauchy in \(H\) exactly when the partial sums of \(\sum |c_k|^2\) are Cauchy in \(\mathbb{R}\). Since \(H\) is complete, the series converges exactly when \(\sum |c_k|^2 < \infty\). The norm identity and the formula for the coefficients follow by continuity of the norm and the inner product from the corresponding identities for partial sums. \(\blacksquare\)

Combining the two theorems, for every \(x\) the series \(\sum_k \langle x, e_k \rangle e_k\) converges, and its sum is the orthogonal projection of \(x\) onto the closed span of the \(e_k\): the difference \(x - \sum_k \langle x, e_k \rangle e_k\) is orthogonal to every \(e_j\). Whether the sum equals \(x\) depends on whether the orthonormal set is large enough.

Orthonormal bases

Theorem 5.3. For an orthonormal sequence \((e_k)\) in \(H\) the following are equivalent. (a) The linear span of the \(e_k\) is dense in \(H\). (b) If \(\langle x, e_k \rangle = 0\) for all \(k\), then \(x = 0\). (c) For every \(x\), \(x = \sum_k \langle x, e_k \rangle e_k\). (d) Parseval's identity: \(\|x\|^2 = \sum_k |\langle x, e_k \rangle|^2\) for every \(x\). (e) \(\langle x, y \rangle = \sum_k \langle x, e_k \rangle \overline{\langle y, e_k \rangle}\) for all \(x, y\).

Proof. (a) implies (b) by Corollary 4.7: a vector orthogonal to a dense subspace is zero. (b) implies (c): the difference \(z = x - \sum_k \langle x, e_k \rangle e_k\) is orthogonal to every \(e_j\), so \(z = 0\). (c) implies (e) by continuity of the inner product applied to partial sums, and (e) implies (d) by taking \(y = x\). (d) implies (a): \(\|x - P_n x\|^2 = \|x\|^2 - \sum_{k \le n} |\langle x, e_k \rangle|^2 \to 0\), so \(x\) is a limit of elements of the span. \(\blacksquare\)

An orthonormal sequence with these properties is an orthonormal basis. The word basis here does not mean a Hamel basis: \(x\) is an infinite sum, convergent in norm, not a finite combination. The definition and theorem extend to uncountable orthonormal families, with sums over the countably many nonzero terms.

The Gram–Schmidt process turns any linearly independent sequence \(v_1, v_2, \ldots\) into an orthonormal sequence with the same partial spans: set \(u_n = v_n - \sum_{k < n} \langle v_n, e_k \rangle e_k\) and \(e_n = u_n / \|u_n\|\). Independence guarantees \(u_n \ne 0\). Theorem 5.4. Every separable Hilbert space has a countable orthonormal basis. Proof: take a countable dense set, discard vectors that are linear combinations of earlier ones, and apply Gram–Schmidt; the resulting orthonormal sequence has the same span, which is dense. \(\blacksquare\) Every Hilbert space, separable or not, has an orthonormal basis: by Zorn's lemma there is a maximal orthonormal family, and maximality is condition (b).

Theorem 5.5. Every infinite-dimensional separable Hilbert space \(H\) is unitarily isomorphic to \(\ell^2\): there is a linear bijection \(U : H \to \ell^2\) with \(\langle Ux, Uy \rangle = \langle x, y \rangle\). Proof: with an orthonormal basis \((e_k)\), let \(Ux = (\langle x, e_k \rangle)_k\). By Parseval it is an isometry into \(\ell^2\), it preserves inner products by (e), and it is onto by Theorem 5.2. \(\blacksquare\) This is the content of the Riesz–Fischer theorem of 1907 in its original form: the map from \(L^2[a, b]\) to the sequence of Fourier coefficients is onto \(\ell^2\). Up to isomorphism there is therefore only one infinite-dimensional separable Hilbert space, and the many concrete realizations of it, \(L^2\) of intervals, of the circle, of \(\mathbb{R}^n\), Sobolev spaces, are distinguished by the operators that act naturally on them, not by their Hilbert space structure.

Legendre polynomials

Apply Gram–Schmidt to \(1, x, x^2, \ldots\) in \(L^2[-1, 1]\). The polynomials obtained, rescaled so that their value at \(1\) is \(1\), are the Legendre polynomials \(P_0 = 1\), \(P_1 = x\), \(P_2 = \frac{1}{2}(3x^2 - 1)\), \(P_3 = \frac{1}{2}(5x^3 - 3x)\), and in general Rodrigues' formula \(P_n(x) = \frac{1}{2^n n!} \frac{d^n}{dx^n} (x^2 - 1)^n\). Orthogonality follows by integrating by parts \(n\) times against any polynomial of lower degree, and \(\int_{-1}^{1} P_n^2 = \frac{2}{2n + 1}\), so \(\sqrt{(2n + 1)/2} \, P_n\) is orthonormal. Since polynomials are dense in \(C[-1, 1]\) by Weierstrass and continuous functions are dense in \(L^2[-1, 1]\) by MATH-441, the span is dense, and the normalized Legendre polynomials form an orthonormal basis. Adrien-Marie Legendre introduced them in 1782 while studying the gravitational attraction of solids of revolution.

The trigonometric system

In \(L^2(-\pi, \pi)\) the functions \(\varepsilon_n(x) = e^{inx} / \sqrt{2\pi}\), \(n \in \mathbb{Z}\), are orthonormal: \(\int_{-\pi}^{\pi} e^{i(n - m)x} \, dx = 0\) for \(n \ne m\). The Fourier coefficients \(\hat{f}(n) = \frac{1}{2\pi} \int_{-\pi}^{\pi} f(x) e^{-inx} \, dx\) differ from \(\langle f, \varepsilon_n \rangle\) by the factor \(\sqrt{2\pi}\), and the partial sums \(S_N f(x) = \sum_{|n| \le N} \hat{f}(n) e^{inx}\) are the orthogonal projections onto the trigonometric polynomials of degree at most \(N\).

Theorem 5.6. The trigonometric system is an orthonormal basis of \(L^2(-\pi, \pi)\). Consequently \(S_N f \to f\) in \(L^2\) for every \(f \in L^2\), and \(\frac{1}{2\pi} \int_{-\pi}^{\pi} |f|^2 = \sum_{n \in \mathbb{Z}} |\hat{f}(n)|^2\).

Proof. By Theorem 5.3 it suffices to show trigonometric polynomials are dense. Step 1 (Fejér's theorem, 1900): for continuous \(2\pi\)-periodic \(g\), the averages \(\sigma_N g = \frac{1}{N + 1} \sum_{k=0}^{N} S_k g\) converge to \(g\) uniformly. Indeed \(\sigma_N g(x) = \frac{1}{2\pi} \int_{-\pi}^{\pi} g(x - t) F_N(t) \, dt\) with the Fejér kernel \(F_N(t) = \frac{1}{N + 1} \Big(\frac{\sin((N + 1)t/2)}{\sin(t/2)}\Big)^2\), which is nonnegative, has \(\frac{1}{2\pi} \int F_N = 1\), and satisfies \(F_N(t) \le \frac{1}{(N + 1) \sin^2(\delta/2)}\) for \(\delta \le |t| \le \pi\). Given \(\varepsilon\), uniform continuity gives \(\delta\) with \(|g(x - t) - g(x)| < \varepsilon\) for \(|t| < \delta\), and then \[|\sigma_N g(x) - g(x)| \le \frac{1}{2\pi} \int_{-\pi}^{\pi} |g(x - t) - g(x)| F_N(t) \, dt \le \varepsilon + \frac{2\|g\|_\infty}{(N + 1)\sin^2(\delta / 2)},\] which is less than \(2\varepsilon\) for large \(N\). Step 2: continuous periodic functions are dense in \(L^2(-\pi, \pi)\): by MATH-441 continuous functions with compact support in \((-\pi, \pi)\) are dense, and these are periodic. Step 3: uniform convergence on a set of finite measure implies \(L^2\) convergence, since \(\|h\|_2 \le \sqrt{2\pi} \|h\|_\infty\). So trigonometric polynomials are dense in \(L^2\). \(\blacksquare\)

Parseval's identity evaluates series. For \(f(x) = x\) on \((-\pi, \pi)\), integration by parts gives \(\hat{f}(n) = \frac{i(-1)^n}{n}\) for \(n \ne 0\) and \(\hat{f}(0) = 0\). Parseval says \(\frac{1}{2\pi} \int_{-\pi}^{\pi} x^2 \, dx = \frac{\pi^2}{3} = \sum_{n \ne 0} \frac{1}{n^2} = 2 \sum_{n=1}^{\infty} \frac{1}{n^2}\), which is Euler's sum \(\sum 1/n^2 = \pi^2/6\).

Norm convergence versus pointwise convergence

Theorem 5.6 is a statement about convergence in the \(L^2\) norm. Pointwise convergence is a different and much harder matter. For the square wave \(f = \operatorname{sgn}(x)\) on \((-\pi, \pi)\), the Fourier series is \(\frac{4}{\pi} \sum_{k \ge 0} \frac{\sin((2k + 1)x)}{2k + 1}\). It converges to \(f\) at every point except the jumps, but the partial sums overshoot near each jump by an amount that does not shrink: the maximum of \(S_N f\) near \(0\) tends to \(\frac{2}{\pi} \int_0^\pi \frac{\sin t}{t} \, dt \approx 1.17898\), so the overshoot is about \(0.0895\) of the total jump of \(2\). Henry Wilbraham described this in 1848, and J. Willard Gibbs rediscovered it in letters to Nature in 1898 and 1899; Maxime Bôcher named it the Gibbs phenomenon in 1906.

The story at general points is subtle. Paul du Bois-Reymond constructed in 1873 a continuous function whose Fourier series diverges at a point, and lesson 8 deduces the existence of such functions from the uniform boundedness principle. Andrey Kolmogorov constructed in 1923 an integrable function whose Fourier series diverges almost everywhere, and later everywhere. The question for \(L^2\) remained open until Lennart Carleson proved in 1966 that the Fourier series of every \(L^2\) function converges almost everywhere, a deep theorem whose proof lies far beyond Hilbert space methods. In contrast, convergence in the \(L^2\) norm, the natural notion for a Hilbert space, takes three lines once completeness of the system is known.

# Gram-Schmidt on 1, x, x^2, ... in L^2[-1, 1] with exact arithmetic, and Fourier partial sums of a square wave.
from fractions import Fraction
import math

def ip(p, q):
    """inner product of polynomials given as coefficient lists, over [-1, 1]"""
    r = [Fraction(0)] * (len(p) + len(q) - 1)
    for i, a in enumerate(p):
        for j, b in enumerate(q):
            r[i + j] += a * b
    return sum(c * Fraction(2, k + 1) for k, c in enumerate(r) if k % 2 == 0)

basis = []
for n in range(5):
    v = [Fraction(0)] * n + [Fraction(1)]               # the monomial x^n
    for u in basis:                                     # subtract projections (orthogonal, not normalized)
        c = ip(v, u) / ip(u, u)
        v = [a - c * (u[k] if k < len(u) else 0) for k, a in enumerate(v)]
    basis.append(v)
for n, v in enumerate(basis):
    scale = sum(v)                                      # value at x = 1
    P = [c / scale for c in v]
    print(f"P_{n}: coefficients {[str(c) for c in P]},  integral of P_n^2 = {ip(P, P)}")

# square wave sgn(x): partial sums (4/pi) * sum over odd m <= N of sin(m x)/m
for N in (9, 39, 159):
    xs = [k * math.pi / (40 * N) for k in range(1, 400)]
    peak = max((4 / math.pi) * sum(math.sin(m * x) / m for m in range(1, N + 1, 2)) for x in xs)
    tail = 1 - (8 / math.pi ** 2) * sum(1 / m ** 2 for m in range(1, N + 1, 2))   # squared L2 error / (2 pi) by Parseval
    print(f"N = {N:3d}: peak of partial sum = {peak:.5f},  normalized squared L2 error = {tail:.5f}")
print("limit of the peak: (2/pi) * Si(pi) =", round(2 / math.pi * sum(math.sin(t) / t * math.pi / 100000 for t in [(k + 0.5) * math.pi / 100000 for k in range(100000)]), 5))

P_0: coefficients ['1'],  integral of P_n^2 = 2
P_1: coefficients ['0', '1'],  integral of P_n^2 = 2/3
P_2: coefficients ['-1/2', '0', '3/2'],  integral of P_n^2 = 2/5
P_3: coefficients ['0', '-3/2', '0', '5/2'],  integral of P_n^2 = 2/7
P_4: coefficients ['3/8', '0', '-15/4', '0', '35/8'],  integral of P_n^2 = 2/9
N =   9: peak of partial sum = 1.18233,  normalized squared L2 error = 0.04040
N =  39: peak of partial sum = 1.17919,  normalized squared L2 error = 0.01013
N = 159: peak of partial sum = 1.17895,  normalized squared L2 error = 0.00253
limit of the peak: (2/pi) * Si(pi) = 1.17898

The output shows the two kinds of convergence side by side. The peak of the partial sum barely moves as \(N\) grows from \(9\) to \(159\), while the normalized squared error \(1 - \frac{8}{\pi^2} \sum_{m \text{ odd}, m \le N} \frac{1}{m^2}\), computed from Parseval's identity, falls roughly like \(\frac{1}{N}\). The overshoot occupies an interval of width about \(\frac{\pi}{N}\), so it contributes less and less to the integral of the squared error.