Differentiation in Rⁿ: the derivative as a linear map

For a function of one variable the derivative \(f^\prime(a)\) is a number, and the definition \(f^\prime(a) = \lim_{h \to 0} \frac{f(a + h) - f(a)}{h}\) cannot be copied to a function \(f: \mathbb{R}^n \to \mathbb{R}^m\), since we cannot divide by a vector \(h\). The way forward is to read the one-variable definition differently. It says that \[f(a + h) = f(a) + f^\prime(a)h + r(h), \quad \text{where } \frac{r(h)}{h} \to 0,\] so the map \(h \mapsto f^\prime(a)h\), multiplication by a number, is the best linear approximation to the change \(f(a + h) - f(a)\). In several variables the derivative is defined to be that best linear approximation, which is now a linear map from \(\mathbb{R}^n\) to \(\mathbb{R}^m\).

Linear maps and their norms

We use the Euclidean norm \(|x| = \sqrt{\sum x_i^2}\) on \(\mathbb{R}^n\); by lesson 9 any other norm would give the same notions of limit and continuity. A linear map \(A: \mathbb{R}^n \to \mathbb{R}^m\) is given by an \(m \times n\) matrix, and we use the same letter for both. Its operator norm is \[\|A\| = \sup\{|Ax| : |x| \le 1\},\] the largest factor by which \(A\) stretches a vector. It is finite, since lesson 9 showed \(|Ax| \le \sqrt{\sum a_{ij}^2}\,|x|\), and it satisfies \(|Ax| \le \|A\||x|\) for all \(x\), \(\|A + B\| \le \|A\| + \|B\|\) and \(\|AB\| \le \|A\|\|B\|\). For instance, the diagonal matrix with entries 3 and \(-5\) has operator norm 5. The set of \(m \times n\) matrices with the metric \(\|A - B\|\) is just \(\mathbb{R}^{mn}\) with an unusual norm, so by lesson 9 it has the same convergent sequences as with the Euclidean norm on the entries.

The definition

Definition. Let \(U \subseteq \mathbb{R}^n\) be open and \(f: U \to \mathbb{R}^m\). Then \(f\) is differentiable at \(a \in U\) if there is a linear map \(T: \mathbb{R}^n \to \mathbb{R}^m\) such that \[\lim_{h \to 0} \frac{|f(a + h) - f(a) - Th|}{|h|} = 0.\] The map \(T\) is the derivative of \(f\) at \(a\), written \(Df(a)\) or \(f^\prime(a)\). We say \(f\) is differentiable on \(U\) if it is differentiable at every point of \(U\).

The limit is a limit in \(\mathbb{R}^n\) as \(h \to 0\) through all nonzero vectors with \(a + h \in U\), and openness of \(U\) guarantees that all small \(h\) are allowed. Writing \(r(h) = f(a + h) - f(a) - Th\), the condition is \(|r(h)| = o(|h|)\): the error of the linear approximation is small compared with \(|h|\).

Theorem 10.1. The derivative, if it exists, is unique, and a function differentiable at \(a\) is continuous at \(a\).

Proof. Suppose \(T\) and \(S\) both satisfy the definition. For a fixed unit vector \(v\) and small \(t > 0\), put \(h = tv\). Then \[|Tv - Sv| = \frac{|Th - Sh|}{t} \le \frac{|f(a + h) - f(a) - Sh|}{|h|} + \frac{|f(a + h) - f(a) - Th|}{|h|} \to 0\] as \(t \to 0\), so \(Tv = Sv\). As \(v\) was any unit vector and both maps are linear, \(T = S\). For continuity, \(|f(a + h) - f(a)| \le |Th| + |r(h)| \le \|T\||h| + |r(h)| \to 0\). \(\square\)

The idea goes back to Otto Stolz, who in 1893 defined the total differential of a function of several variables by this kind of approximation. W. H. Young's tract of 1910 on the fundamental theorems of the differential calculus made the linear approximation the central definition, and Maurice Fréchet, in papers of 1911 and 1912, stated it for maps between more general spaces, which is why the derivative of this lesson is often called the Fréchet derivative.

Partial derivatives and the Jacobian matrix

For \(f: U \to \mathbb{R}^m\) with components \(f_1, \dots, f_m\), the partial derivative \(\frac{\partial f_i}{\partial x_j}(a)\) is the ordinary derivative at 0 of \(t \mapsto f_i(a + te_j)\), where \(e_j\) is the \(j\)-th standard basis vector. More generally, the directional derivative in the direction \(v\) is \(D_vf(a) = \lim_{t \to 0} \frac{f(a + tv) - f(a)}{t}\).

Theorem 10.2. If \(f\) is differentiable at \(a\), then every directional derivative exists and \(D_vf(a) = Df(a)v\). In particular the matrix of \(Df(a)\) is the Jacobian matrix \[Df(a) = \Big(\frac{\partial f_i}{\partial x_j}(a)\Big)_{1 \le i \le m,\ 1 \le j \le n}.\]

Proof. For \(v \ne 0\) and \(h = tv\), \(\big|\frac{f(a + tv) - f(a)}{t} - Df(a)v\big| = |v|\frac{|r(tv)|}{|tv|} \to 0\) as \(t \to 0\). The \(j\)-th column of the matrix of \(Df(a)\) is \(Df(a)e_j = D_{e_j}f(a)\), whose \(i\)-th entry is \(\frac{\partial f_i}{\partial x_j}(a)\). \(\square\)

For a real-valued \(f\), the \(1 \times n\) matrix \(Df(a)\) is the row of partial derivatives, and the vector with the same entries is the gradient \(\nabla f(a)\), so \(Df(a)h = \nabla f(a) \cdot h\). By Cauchy–Schwarz, the directional derivative \(\nabla f(a) \cdot v\) over unit vectors \(v\) is largest when \(v\) points along \(\nabla f(a)\), and its largest value is \(|\nabla f(a)|\). The name Jacobian honors Carl Gustav Jacobi, whose 1841 memoir on functional determinants studied the determinant of this matrix.

The converse of theorem 10.2 is false: partial derivatives, and even all directional derivatives, can exist without the function being differentiable.

What rescues the situation is continuity of the partial derivatives.

Theorem 10.3. Let \(U \subseteq \mathbb{R}^n\) be open and \(f: U \to \mathbb{R}^m\). If all partial derivatives \(\frac{\partial f_i}{\partial x_j}\) exist on \(U\) and are continuous at \(a\), then \(f\) is differentiable at \(a\).

Proof. A vector-valued map is differentiable if and only if each component is, since the Euclidean norm of a vector is at most the sum of the absolute values of its components. So assume \(m = 1\). Choose \(r > 0\) with \(B(a, r) \subseteq U\) and let \(|h| < r\), \(h = \sum_j h_je_j\). Define \(p_0 = a\) and \(p_j = a + h_1e_1 + \dots + h_je_j\), so \(p_n = a + h\). Each \(p_j\), and every point of the segment from \(p_{j-1}\) to \(p_j\), is within \(|h|\) of \(a\), so lies in the ball. Then \[f(a + h) - f(a) = \sum_{j=1}^{n} (f(p_j) - f(p_{j-1})).\] Along the segment from \(p_{j-1}\) to \(p_j\) only the \(j\)-th coordinate changes, so the one-variable mean value theorem gives a point \(q_j\) on it with \(f(p_j) - f(p_{j-1}) = h_j\frac{\partial f}{\partial x_j}(q_j)\). Hence \[\Big|f(a + h) - f(a) - \sum_j h_j\frac{\partial f}{\partial x_j}(a)\Big| \le \sum_j |h_j|\Big|\frac{\partial f}{\partial x_j}(q_j) - \frac{\partial f}{\partial x_j}(a)\Big| \le |h|\sum_j \Big|\frac{\partial f}{\partial x_j}(q_j) - \frac{\partial f}{\partial x_j}(a)\Big|.\] As \(h \to 0\), each \(q_j \to a\), and by continuity of the partials at \(a\) the last sum tends to 0. So \(f\) is differentiable at \(a\), with derivative given by the partial derivatives. \(\square\)

A map whose partial derivatives exist and are continuous on \(U\) is called continuously differentiable, or \(C^1\). By theorem 10.3 it is differentiable, and \(x \mapsto Df(x)\) is continuous from \(U\) to the space of matrices. Polynomials, exponentials, trigonometric functions and their compositions are \(C^1\) wherever they are defined, so in practice differentiability is checked by computing partial derivatives and observing that they are continuous.

The chain rule

Theorem 10.4 (chain rule). Let \(f: U \to \mathbb{R}^m\) be differentiable at \(a\), with \(U \subseteq \mathbb{R}^n\) open, and let \(g: V \to \mathbb{R}^p\) be differentiable at \(b = f(a)\), with \(V \subseteq \mathbb{R}^m\) open and \(f(U) \subseteq V\). Then \(g \circ f\) is differentiable at \(a\) and \[D(g \circ f)(a) = Dg(b)\,Df(a).\]

Proof. Let \(A = Df(a)\) and \(B = Dg(b)\). Write \(f(a + h) = b + Ah + r(h)\) with \(\frac{|r(h)|}{|h|} \to 0\), and \(g(b + k) = g(b) + Bk + s(k)\), where \(|s(k)| \le \eta(k)|k|\) with \(\eta(k) \to 0\) as \(k \to 0\) and \(\eta(0) = 0\). Put \(k = k(h) = f(a + h) - b = Ah + r(h)\). Then \[g(f(a + h)) - g(b) - BAh = Br(h) + s(k).\] The first term satisfies \(|Br(h)| \le \|B\||r(h)| = o(|h|)\). For the second, \(|k| \le \|A\||h| + |r(h)| \le (\|A\| + 1)|h|\) once \(h\) is small, and \(k \to 0\) as \(h \to 0\), so \(|s(k)| \le \eta(k)(\|A\| + 1)|h| = o(|h|)\). Hence \(g \circ f\) is differentiable at \(a\) with derivative \(BA\). \(\square\)

In matrix terms the chain rule multiplies Jacobian matrices, and written out entry by entry it is the familiar formula \(\frac{\partial (g \circ f)_i}{\partial x_j} = \sum_k \frac{\partial g_i}{\partial y_k}\frac{\partial f_k}{\partial x_j}\). The linear map point of view makes the rule almost obvious: the best linear approximation of a composition is the composition of the best linear approximations.

The mean value inequality

The mean value theorem has no exact analog for vector-valued maps. The map \(f(t) = (\cos t, \sin t)\) has \(f(2\pi) - f(0) = 0\), but \(f^\prime(t) = (-\sin t, \cos t)\) is never 0, so no \(c\) satisfies \(f(2\pi) - f(0) = 2\pi f^\prime(c)\). What survives is an inequality, and it is all we need.

Theorem 10.5 (mean value inequality). Let \(f: U \to \mathbb{R}^m\) be differentiable on an open set \(U\) containing the segment from \(a\) to \(b\), and suppose \(\|Df(x)\| \le M\) at every point \(x\) of the segment. Then \(|f(b) - f(a)| \le M|b - a|\).

Proof. Let \(v = f(b) - f(a)\); if \(v = 0\) there is nothing to prove. Define \(\phi(t) = v \cdot f(a + t(b - a))\) on an open interval containing \([0, 1]\). By the chain rule \(\phi\) is differentiable with \(\phi^\prime(t) = v \cdot Df(a + t(b - a))(b - a)\). The one-variable mean value theorem gives \(\tau \in (0, 1)\) with \[|v|^2 = \phi(1) - \phi(0) = \phi^\prime(\tau) \le |v|\,\|Df(c)\|\,|b - a| \le |v|M|b - a|,\] where \(c = a + \tau(b - a)\) and we used Cauchy–Schwarz. Dividing by \(|v|\) gives the result. \(\square\)

A set is convex if it contains the segment between any two of its points; balls are convex. On a convex open set where \(\|Df\| \le M\), the inequality makes \(f\) Lipschitz with constant \(M\). In particular, if \(Df = 0\) on a convex open set, \(f\) is constant there. Lesson 11 applies the inequality to show that a certain map is a contraction.

Computing derivatives numerically

Since \(Df(a)e_j = \lim_{t \to 0} \frac{f(a + te_j) - f(a)}{t}\), the columns of the Jacobian can be approximated by difference quotients. The forward difference \(\frac{f(a + te_j) - f(a)}{t}\) has error of order \(t\) for smooth \(f\), and the central difference \(\frac{f(a + te_j) - f(a - te_j)}{2t}\) has error of order \(t^2\), by Taylor's theorem. In floating point arithmetic, rounding errors of about \(10^{-16}\) are divided by \(t\), so making \(t\) too small makes things worse. The code compares both formulas with the exact Jacobian of \(f(x, y) = (x^2y, \ e^x\sin y, \ x + y^3)\) at \((1, 0.5)\).

import math

def f(p):
    x, y = p
    return [x * x * y, math.exp(x) * math.sin(y), x + y ** 3]

def exact_J(p):
    x, y = p
    return [[2 * x * y, x * x],
            [math.exp(x) * math.sin(y), math.exp(x) * math.cos(y)],
            [1.0, 3 * y * y]]

def jacobian(f, p, t, central=True):
    n, fp = len(p), f(p)
    cols = []
    for j in range(n):
        up = list(p); up[j] += t
        if central:
            dn = list(p); dn[j] -= t
            col = [(a - b) / (2 * t) for a, b in zip(f(up), f(dn))]
        else:
            col = [(a - b) / t for a, b in zip(f(up), fp)]
        cols.append(col)
    return [[cols[j][i] for j in range(n)] for i in range(len(fp))]   # rows = components

def max_err(A, B):
    return max(abs(a - b) for ra, rb in zip(A, B) for a, b in zip(ra, rb))

p = (1.0, 0.5)
J = exact_J(p)
print("exact Jacobian rows:", [[round(v, 6) for v in row] for row in J])
print("    t        forward error   central error")
for k in range(1, 13, 2):
    t = 10.0 ** -k
    print(f"1e-{k:02d}     {max_err(jacobian(f, p, t, False), J):12.3e}    {max_err(jacobian(f, p, t), J):12.3e}")

exact Jacobian rows: [[1.0, 1.0], [1.303214, 2.385517], [1.0, 0.75]]
    t        forward error   central error
1e-01        1.600e-01       1.000e-02
1e-03        1.501e-03       1.000e-06
1e-05        1.500e-05       1.048e-10
1e-07        1.487e-07       1.227e-09
1e-09        8.274e-08       8.274e-08
1e-11        1.047e-05       2.722e-06

The forward error falls by a factor of about 100 each time \(t\) shrinks by 100 until \(t\) is near \(10^{-7}\) or \(10^{-9}\), and the central error falls by about \(10^4\), as Taylor's theorem predicts, until \(t\) is near \(10^{-5}\). Below those values rounding error, which grows like \(\frac{10^{-16}}{t}\), dominates and the approximations get worse. Automatic differentiation, used in modern scientific computing and machine learning, avoids the dilemma by applying the chain rule exactly to the steps of a computation.