Taylor polynomials, approximation and numerical error

Engineers approximate all the time. A pendulum's period is calculated with \(\sin\theta \approx \theta\); the sag of a power line is modeled as a parabola instead of a catenary; a nonlinear sensor is linearized around its operating point; a computer evaluates \(e^x\) by adding a few terms of a polynomial. Every approximation carries an error, and good engineering requires knowing how big that error is. Earlier lessons used the tangent line as an approximation (Lesson 3), finite differences to estimate derivatives from data (Lesson 3), Newton's method to solve equations (Lesson 7), and the trapezoid rule to integrate data (Lesson 9). This lesson develops the idea behind all of them: Taylor polynomials, which approximate a function near a point by matching its derivatives, together with a formula for the error. It then uses that formula to explain how accurate numerical methods are, and closes with the limits imposed by the finite precision of computer arithmetic.

From tangent lines to Taylor polynomials

The linear approximation \(f(x) \approx f(a) + f'(a)(x - a)\) matches the function's value and slope at \(a\). It works well very close to \(a\) but drifts away as the curve bends. A better approximation should also match the bending, that is, the second derivative. The quadratic polynomial that matches \(f\), \(f'\) and \(f''\) at \(a\) is:

Continuing the pattern, the Taylor polynomial of degree \(n\) matches the first \(n\) derivatives:

where \(f^{(k)}\) is the \(k\)-th derivative and \(k! = 1 \cdot 2 \cdots k\) is \(k\) factorial (with \(0! = 1\)). The factorials appear because the \(k\)-th derivative of \((x - a)^k\) is \(k!\). When \(a = 0\), the polynomial is often called a Maclaurin polynomial.

The most frequently used Taylor polynomials, all centered at 0, are:

The last, the binomial approximation, is particularly useful: \(\sqrt{1 + x} \approx 1 + x/2\) and \(1/(1 + x) \approx 1 - x\) for small \(x\).

Engineering approximations

Many standard engineering formulas are Taylor approximations in disguise:

• The small-angle approximation \(\sin\theta \approx \theta\) (with \(\theta\) in radians) gives the simple pendulum period \(T = 2\pi\sqrt{L/g}\) and the linear theory of beam deflection. It is accurate to within 1 percent up to about 14° (0.24 rad), since the next term, \(\theta^3/6\), is then about 1 percent of \(\theta\).
• A cable hanging under its own weight forms a catenary, \(y = a\cosh(x/a)\). Expanding \(\cosh u \approx 1 + u^2/2\) gives a parabola, \(y \approx a + x^2/(2a)\), which is why power line sag calculations use parabolas when the sag is small compared with the span.
• The relativistic and quantum corrections that engineers occasionally meet are usually the second term of a Taylor expansion, which is why they are negligible at ordinary speeds and energies.
• Linearizing a nonlinear device around its operating point, as in transistor amplifiers, valve flow characteristics or the small-signal converter models of ELE-470, uses the first-order Taylor polynomial in each variable.

How big is the error?

Taylor's theorem gives the error of the approximation. If \(f\) has \(n + 1\) derivatives, then for each \(x\) there is a point \(c\) between \(a\) and \(x\) such that:

This is the Lagrange form of the remainder, a generalization of the Mean Value Theorem of Lesson 6 (which is the case \(n = 0\)). We do not know \(c\), but if \(|f^{(n+1)}| \le M\) between \(a\) and \(x\), then the error is at most \(M|x - a|^{n+1}/(n + 1)!\). Two lessons follow. The error shrinks rapidly as \(x\) approaches \(a\), like the \((n + 1)\)-th power of the distance. And the factorial in the denominator makes high-degree polynomials very accurate for well-behaved functions.

Why numerical methods work, and how well

Taylor's theorem explains the accuracy of the numerical methods met in earlier lessons.

Finite differences

Expanding \(f(x + h) = f(x) + hf'(x) + \tfrac{h^2}{2}f''(x) + \tfrac{h^3}{6}f'''(x) + \cdots\) and rearranging shows that the forward difference has an error of about \(\tfrac{h}{2}f''(x)\), proportional to \(h\). Writing the same expansion for \(f(x - h)\) and subtracting cancels the \(h^2\) terms, so the central difference has an error of about \(\tfrac{h^2}{6}f'''(x)\), proportional to \(h^2\). This is why the central difference in Lesson 3 was about fifty times more accurate with the same step: halving \(h\) halves the forward error but quarters the central error. A method whose error is proportional to \(h^p\) is said to be of order \(p\).

Numerical integration

The trapezoid rule is of order 2: its error over a fixed interval is approximately \(-\tfrac{(b - a)h^2}{12}f''\), so halving the step reduces the error by a factor of four. Simpson's rule, which fits parabolas through groups of three points, is of order 4, so halving the step reduces its error by a factor of sixteen. Knowing the order allows a powerful trick, Richardson extrapolation: if \(T_h\) and \(T_{h/2}\) are trapezoid estimates with step \(h\) and \(h/2\), the combination \((4T_{h/2} - T_h)/3\) cancels the leading error term.

Newton's method

Lesson 7 observed that Newton's method roughly doubles the number of correct digits at each step. Taylor's theorem explains why: expanding \(f\) around the current estimate shows that the new error is proportional to the square of the old error, \(e_{n+1} \approx \tfrac{f''(r)}{2f'(r)}e_n^2\). This quadratic convergence holds once the estimate is close to a simple root. Far from the root, or where \(f'\) is small, the method can overshoot, cycle or diverge, which is why practical solvers combine Newton steps with safeguards such as bisection.

Round-off error and floating-point arithmetic

Computers store numbers in floating-point form with a fixed number of significant digits. Standard double-precision arithmetic carries about 16 significant decimal digits; the gap between 1 and the next representable number, called machine epsilon, is about \(2.2 \times 10^{-16}\). Every arithmetic operation rounds its result, introducing a relative error of up to about \(10^{-16}\). Usually this is negligible, but two situations magnify it.

Catastrophic cancellation

Subtracting two nearly equal numbers loses the digits they share. For \(x = 10^{-8}\), \(\cos x = 0.999\,999\,999\,999\,999\,95\), which rounds to exactly 1 in double precision, so computing \(1 - \cos x\) gives 0 instead of the correct \(5 \times 10^{-17}\). Rewriting the expression avoids the subtraction: \(1 - \cos x = 2\sin^2(x/2)\), which computes accurately. The Taylor polynomial \(x^2/2\) gives the answer too. The same problem appears in the quadratic formula when \(b^2 \gg 4ac\), in computing small differences of large measured quantities, and in variance calculations.

Truncation versus round-off

In a finite difference, the truncation error (from the approximation) shrinks as \(h\) gets smaller, but the round-off error grows, because the numerator \(f(x + h) - f(x)\) is a difference of nearly equal numbers, each carrying a rounding error of about \(\varepsilon |f|\), divided by a small \(h\). The total error is roughly \(\tfrac{h}{2}|f''| + \tfrac{2\varepsilon |f|}{h}\), smallest when \(h\) is about \(\sqrt{\varepsilon}\), around \(10^{-8}\) for a forward difference in double precision. Making \(h\) smaller than this makes the answer worse, not better. Measured data, whose noise is far larger than machine epsilon, has a much larger optimal step, which is why noisy data must be smoothed before differentiation.

Approximation as an engineering habit

The tools of this lesson support habits that every engineer needs. Before trusting a computed result, estimate it roughly by hand using linear or quadratic approximations. When using a numerical method, check convergence by repeating the calculation with a smaller step or tighter tolerance and confirming that the answer settles down at the expected rate. When a formula involves subtracting nearly equal quantities, look for a rearrangement. And when an approximation such as the small-angle formula is used, check the size of the first neglected term: if it is small compared with the required accuracy, the approximation is safe.