Bounded linear operators and dual spaces

Lesson 1 showed that a linear map between infinite-dimensional normed spaces need not be continuous. This lesson identifies the continuous ones: they are exactly the bounded operators, those that stretch vectors by at most a fixed factor. The smallest such factor is the operator norm, which turns the set of bounded operators into a normed space, complete when the target is. We compute operator norms for matrices and integral operators, prove that the Neumann series inverts small perturbations of the identity, and study the most important special case: the dual space of continuous linear functionals, which we identify concretely for the sequence spaces.

Bounded equals continuous

Definition 3.1. A linear map \(T : X \to Y\) between normed spaces is bounded if there is a constant \(C\) with \(\|Tx\| \le C \|x\|\) for all \(x \in X\). The operator norm is \[\|T\| = \sup_{x \ne 0} \frac{\|Tx\|}{\|x\|} = \sup_{\|x\| \le 1} \|Tx\| = \sup_{\|x\| = 1} \|Tx\|,\] the least admissible \(C\). The set of bounded linear maps is written \(B(X, Y)\), and \(B(X) = B(X, X)\).

The word bounded does not mean that the image of \(T\) is a bounded set, which only happens for \(T = 0\). It means that \(T\) maps bounded sets to bounded sets, and in particular the unit ball into the ball of radius \(\|T\|\).

Theorem 3.2. For a linear map \(T : X \to Y\) the following are equivalent: (a) \(T\) is continuous; (b) \(T\) is continuous at \(0\); (c) \(T\) is bounded. In that case \(T\) is Lipschitz with constant \(\|T\|\).

Proof. (a) implies (b) trivially. (b) implies (c): with \(\varepsilon = 1\), continuity at \(0\) gives \(\delta > 0\) such that \(\|x\| \le \delta\) implies \(\|Tx\| \le 1\). For \(x \ne 0\), the vector \(\delta x / \|x\|\) has norm \(\delta\), so \(\|T(\delta x / \|x\|)\| \le 1\), that is, \(\|Tx\| \le \delta^{-1} \|x\|\). (c) implies (a): \(\|Tx - Ty\| = \|T(x - y)\| \le \|T\| \, \|x - y\|\). \(\blacksquare\)

Linearity converts local information at one point into global information everywhere, which is why a single estimate at the origin suffices. For linear functionals there is a further useful criterion. Proposition 3.3. A linear functional \(f : X \to \mathbb{K}\) is continuous if and only if its kernel is closed. Proof: continuity gives a closed kernel as the preimage of \(\{0\}\). Conversely suppose \(f\) is unbounded but \(\ker f\) is closed. Then \(f \ne 0\); choose \(x_0\) with \(f(x_0) = 1\), and \(x_n\) with \(\|x_n\| \le \frac{1}{n}\) and \(f(x_n) = 1\), possible because \(f\) is unbounded. Then \(x_0 - x_n \in \ker f\) and \(x_0 - x_n \to x_0\), so \(x_0 \in \ker f\), a contradiction. \(\blacksquare\) In fact the kernel of a discontinuous functional is dense.

Computing operator norms

For a matrix \(A = (a_{ij})\) acting from \(\mathbb{K}^n\) to \(\mathbb{K}^m\), the operator norm depends on the norms chosen. With the \(1\)-norm on both sides, \(\|A\| = \max_j \sum_i |a_{ij}|\), the largest column sum. With the supremum norm on both sides, \(\|A\| = \max_i \sum_j |a_{ij}|\), the largest row sum. With the Euclidean norm, \(\|A\|\) is the largest singular value, the square root of the largest eigenvalue of \(A^* A\). Proof of the first: \(\|Ax\|_1 = \sum_i |\sum_j a_{ij} x_j| \le \sum_j |x_j| \sum_i |a_{ij}| \le (\max_j \sum_i |a_{ij}|) \|x\|_1\), and equality holds at the basis vector \(e_j\) of the largest column. The second is similar, with equality at the vector of signs of the largest row.

Integral operators. Let \(k\) be continuous on \([0, 1]^2\) and define \((Ku)(x) = \int_0^1 k(x, y) u(y) \, dy\). Uniform continuity of \(k\) shows \(Ku\) is continuous, and \[|Ku(x)| \le \int_0^1 |k(x, y)| \, |u(y)| \, dy \le \Big(\max_x \int_0^1 |k(x, y)| \, dy\Big) \|u\|_\infty,\] so on \(C[0, 1]\) with the supremum norm, \(\|K\| \le \max_x \int_0^1 |k(x, y)| \, dy\). Equality holds: if the maximum is attained at \(x_0\), approximate the discontinuous function \(\overline{\operatorname{sgn} k(x_0, y)}\) by continuous functions of norm at most \(1\), so that \(Ku(x_0)\) comes arbitrarily close to \(\int |k(x_0, y)| \, dy\). On \(L^2[0, 1]\) the Cauchy–Schwarz inequality gives instead the Hilbert–Schmidt bound \(\|K\| \le (\iint |k|^2)^{1/2}\), which is usually not sharp.

Multiplication and shift operators. On \(\ell^p\), the diagonal operator \(D_\lambda x = (\lambda_1 x_1, \lambda_2 x_2, \ldots)\) is bounded exactly when \(\lambda\) is bounded, and \(\|D_\lambda\| = \sup |\lambda_n|\). On \(L^p(\mu)\), multiplication by a bounded measurable \(g\) has norm \(\|g\|_\infty\), the essential supremum. The right and left shifts on \(\ell^p\) have norm \(1\). The differentiation operator from lesson 1 is unbounded on \((C^1[0, 1], \|\cdot\|_\infty)\) but bounded, with norm at most \(1\), from \((C^1[0, 1], \|\cdot\|_{C^1})\) to \((C[0, 1], \|\cdot\|_\infty)\). Boundedness is a property of an operator together with the norms on its domain and target.

The space of bounded operators

Sums and scalar multiples of bounded operators are bounded, and the operator norm satisfies the norm axioms, so \(B(X, Y)\) is a normed space. Composition satisfies \(\|ST\| \le \|S\| \, \|T\|\), since \(\|STx\| \le \|S\| \, \|Tx\| \le \|S\| \, \|T\| \, \|x\|\). In particular \(\|T^n\| \le \|T\|^n\).

Theorem 3.4. If \(Y\) is a Banach space, then \(B(X, Y)\) is a Banach space.

Proof. Let \((T_n)\) be Cauchy in \(B(X, Y)\). For each \(x\), \(\|T_n x - T_m x\| \le \|T_n - T_m\| \, \|x\|\), so \((T_n x)\) is Cauchy in \(Y\) and converges to a vector we call \(Tx\). Limits of linear expressions are linear, so \(T\) is linear. Given \(\varepsilon > 0\) choose \(N\) with \(\|T_n - T_m\| < \varepsilon\) for \(m, n \ge N\). Then \(\|T_n x - T_m x\| \le \varepsilon \|x\|\), and letting \(m \to \infty\), \(\|T_n x - Tx\| \le \varepsilon \|x\|\) for all \(x\) and \(n \ge N\). Thus \(T_N - T\) is bounded, so \(T = T_N - (T_N - T)\) is bounded, and \(\|T_n - T\| \le \varepsilon\) for \(n \ge N\). \(\blacksquare\)

Only the completeness of \(Y\) was used; \(X\) can be any normed space. Taking \(Y = \mathbb{K}\), the dual space \(X^* = B(X, \mathbb{K})\) is always a Banach space, even when \(X\) is not complete.

The Neumann series

Theorem 3.5. Let \(X\) be a Banach space and \(T \in B(X)\) with \(\|T\| < 1\). Then \(I - T\) is invertible in \(B(X)\), \[(I - T)^{-1} = \sum_{k=0}^{\infty} T^k, \qquad \|(I - T)^{-1}\| \le \frac{1}{1 - \|T\|}.\]

Proof. Since \(\sum \|T^k\| \le \sum \|T\|^k = \frac{1}{1 - \|T\|}\), the series is absolutely convergent in the Banach space \(B(X)\) and converges to some \(S\) by Theorem 2.4, with \(\|S\|\) bounded as claimed. For the partial sums \(S_n = \sum_{k=0}^{n} T^k\), the telescoping identities \((I - T) S_n = S_n (I - T) = I - T^{n+1}\) hold, and \(\|T^{n+1}\| \le \|T\|^{n+1} \to 0\). Passing to the limit, using continuity of multiplication in \(B(X)\), gives \((I - T)S = S(I - T) = I\). \(\blacksquare\)

This is the geometric series \(\frac{1}{1 - t} = \sum t^k\) for operators, and it is the oldest method for integral equations. Applied to \(u - \lambda K u = f\) with \(|\lambda| \, \|K\| < 1\), it gives the unique solution \(u = \sum_k \lambda^k K^k f\), the Liouville–Neumann series, named for Carl Neumann's use of it in potential theory in the 1870s. Corollary 3.6. The invertible operators form an open subset of \(B(X)\), and inversion is continuous on it. Proof: if \(A\) is invertible and \(\|B - A\| < \|A^{-1}\|^{-1}\), then \(B = A(I - A^{-1}(A - B))\) with \(\|A^{-1}(A - B)\| < 1\), so \(B\) is invertible; the norm estimate on the Neumann series gives continuity. \(\blacksquare\)

# Operator norms of a matrix in three norms, and the Neumann series for an integral equation.
import math

A = [[1.0, 2.0, -3.0], [0.0, 1.0, 0.0], [2.0, 0.0, 1.0]]
n = len(A)
col = max(sum(abs(A[i][j]) for i in range(n)) for j in range(n))
row = max(sum(abs(A[i][j]) for j in range(n)) for i in range(n))
# Euclidean norm: power iteration on A^T A gives the largest singular value
x = [1.0, 1.0, 1.0]
for _ in range(200):
    y = [sum(A[i][j] * x[j] for j in range(n)) for i in range(n)]
    x = [sum(A[i][j] * y[i] for i in range(n)) for j in range(n)]
    s = math.sqrt(sum(t * t for t in x)); x = [t / s for t in x]
Ax = [sum(A[i][j] * x[j] for j in range(n)) for i in range(n)]
print(f"1-norm (column sums) = {col},  sup-norm (row sums) = {row},  2-norm = {math.sqrt(sum(t*t for t in Ax)):.4f}")

# Solve u(x) - lam * int_0^1 exp(-|x - y|) u(y) dy = 1 by the Neumann series on a grid.
N, lam = 400, 0.8
h = 1 / N
xs = [(i + 0.5) * h for i in range(N)]
K = [[math.exp(-abs(a - b)) * h for b in xs] for a in xs]
normK = max(sum(row) for row in K)            # sup-norm operator norm of the discretized K
print(f"discretized |K| = {normK:.4f}, so |lam K| = {lam * normK:.4f} < 1")
f = [1.0] * N
u, term = f[:], f[:]
for k in range(1, 60):
    term = [lam * sum(K[i][j] * term[j] for j in range(N)) for i in range(N)]
    u = [a + b for a, b in zip(u, term)]
    if k in (1, 5, 20, 59):
        print(f"after {k:2d} terms: u(0) = {u[0]:.6f}, u(1/2) = {u[N // 2]:.6f}, size of last term = {max(map(abs, term)):.2e}")

1-norm (column sums) = 4.0,  sup-norm (row sums) = 6.0,  2-norm = 3.7954
discretized |K| = 0.7869, so |lam K| = 0.6296 < 1
after  1 terms: u(0) = 1.506329, u(1/2) = 1.629551, size of last term = 6.30e-01
after  5 terms: u(0) = 2.144574, u(1/2) = 2.432016, size of last term = 7.72e-02
after 20 terms: u(0) = 2.233241, u(1/2) = 2.543535, size of last term = 2.90e-05
after 59 terms: u(0) = 2.233275, u(1/2) = 2.543576, size of last term = 3.59e-14

For the kernel \(e^{-|x - y|}\), the row integral \(\int_0^1 e^{-|x - y|} \, dy = 2 - e^{-x} - e^{x - 1}\) is largest at \(x = \frac{1}{2}\), where it equals \(2 - 2e^{-1/2} \approx 0.787\). So \(\|0.8K\| \approx 0.63\), and each new term of the series is smaller than the previous one by at least that factor, as the output shows.

Dual spaces

Definition 3.7. The dual space of a normed space \(X\) is \(X^* = B(X, \mathbb{K})\), the Banach space of continuous linear functionals with \(\|f\| = \sup_{\|x\| \le 1} |f(x)|\).

Identifying a dual space means finding a concrete space of objects that represent all continuous functionals. For finite-dimensional \(\mathbb{K}^n\) with the \(p\)-norm, every functional has the form \(f(x) = \sum y_i x_i\), and Hölder's inequality with its equality case shows that \(\|f\| = \|y\|_q\), where \(\frac{1}{p} + \frac{1}{q} = 1\). The same is true for sequences, with one exception.

Theorem 3.8. Let \(1 \le p < \infty\) and let \(q\) be the conjugate exponent, with \(q = \infty\) when \(p = 1\). For \(y \in \ell^q\) define \(f_y(x) = \sum_n y_n x_n\). Then \(y \mapsto f_y\) is an isometric isomorphism of \(\ell^q\) onto \((\ell^p)^*\).

Proof. By Hölder's inequality the series converges absolutely and \(|f_y(x)| \le \|y\|_q \|x\|_p\), so \(f_y \in (\ell^p)^*\) with \(\|f_y\| \le \|y\|_q\). The map is clearly linear. Now let \(f \in (\ell^p)^*\) be arbitrary and set \(y_n = f(e_n)\). Case \(p = 1\): \(|y_n| = |f(e_n)| \le \|f\|\), so \(\|y\|_\infty \le \|f\|\). Case \(1 < p < \infty\): fix \(N\) and let \(x_n = |y_n|^{q - 1} \overline{\operatorname{sgn} y_n}\) for \(n \le N\) and \(x_n = 0\) otherwise, where \(\operatorname{sgn} y = y / |y|\) and \(\operatorname{sgn} 0 = 0\). Then \(f(x) = \sum_{n \le N} |y_n|^q\), and since \((q - 1)p = q\), \(\|x\|_p = (\sum_{n \le N} |y_n|^q)^{1/p}\). From \(f(x) \le \|f\| \, \|x\|_p\) we get \((\sum_{n \le N} |y_n|^q)^{1 - 1/p} \le \|f\|\), and \(1 - \frac{1}{p} = \frac{1}{q}\). Letting \(N \to \infty\), \(\|y\|_q \le \|f\|\). In both cases \(y \in \ell^q\), and \(f\) and \(f_y\) agree on every \(e_n\), hence on \(c_{00}\) by linearity. Since \(p < \infty\), \(c_{00}\) is dense in \(\ell^p\), and two continuous functionals that agree on a dense set are equal. So \(f = f_y\) and \(\|f\| = \|y\|_q\). \(\blacksquare\)

The same proof shows that \((c_0)^* = \ell^1\), since \(c_{00}\) is dense in \(c_0\). It fails for \(p = \infty\): \(c_{00}\) is not dense in \(\ell^\infty\), and lesson 7 constructs functionals on \(\ell^\infty\), the Banach limits, that vanish on every \(e_n\) without being zero. Thus \((\ell^\infty)^*\) is strictly larger than \(\ell^1\). For a general measure space, the analogous theorem is \((L^p)^* = L^q\) for \(1 < p < \infty\), and for \(p = 1\) when the measure is \(\sigma\)-finite; its proof uses the Radon–Nikodym theorem from the capstone of MATH-441. For \(C[a, b]\), Riesz proved in 1909 that the dual consists of Stieltjes integrals \(f \mapsto \int f \, dg\) with \(g\) of bounded variation, equivalently the finite signed Borel measures.