Why statistical mechanics: from microscopic rules to macroscopic laws

Thermodynamics, the subject of PHYS-261, is one of the most successful theories in physics, and it is also one of the strangest. Its laws, the conservation of energy, the increase of entropy and the unattainability of absolute zero, apply equally to steam engines, stars, batteries and living cells, and they say nothing at all about what any of those systems is made of. Thermodynamics treats matter as a black box with a handful of dials: temperature, pressure, volume, energy and entropy. Statistical mechanics opens the box. It starts from the microscopic rules, classical or quantum mechanics applied to an enormous number of particles, and derives the thermodynamic laws as statements about averages over those particles. In doing so it explains where the laws come from, why they work so well, when they fail, and how to calculate the quantities that thermodynamics can only measure: heat capacities, equations of state, chemical potentials, magnetizations, equilibrium constants and the temperatures at which phases change.

PHYS-261 already met the central ideas in its last three lessons: the multiplicity of an Einstein solid, Boltzmann's formula \(S = k\ln\Omega\), the definition of temperature as \(1/T = \partial S/\partial U\), the Boltzmann distribution and the partition function. This course builds the subject properly. It sets up the three standard ensembles (microcanonical, canonical and grand canonical) as a logical structure, derives thermodynamics from each, and then uses them to understand real systems: paramagnets, gases of molecules with rotation and vibration, electrons in metals, white dwarf stars, photons, phonons, Bose–Einstein condensates, magnets near their critical points, chemical reactions, adsorbed gases and polymers. The quantum mechanics of PHYS-372, discrete energy levels and the quantum states of a particle in a box, enters constantly, because counting states is the heart of the method.

Two levels of description

Consider a liter of air in a box at room temperature. At the microscopic level it is about \(2.5 \times 10^{22}\) molecules, each with a position, a velocity, a rotational state and a vibrational state. A complete specification of the microstate is a list of perhaps \(10^{23}\) numbers, and that list changes completely every nanosecond as the molecules collide. At the macroscopic level the same gas is described by three numbers, its temperature, pressure and volume, which stay perfectly steady. A macrostate is a specification of the system by a few such quantities. Enormous numbers of microstates correspond to the same macrostate, and this is the first key fact: the macroscopic description throws away almost all of the information, yet it predicts everything that a thermometer or a pressure gauge can measure.

Why does this work? Why is the pressure of the gas steady even though the molecules hitting a wall arrive at random? The answer is the law of large numbers. When a quantity is the sum of many independent random contributions, its average grows in proportion to the number of contributions \(N\), while its fluctuations grow only as \(\sqrt N\). The relative fluctuation therefore falls as \(1/\sqrt N\). For \(N = 10^{22}\) that is \(10^{-11}\), far below what any instrument can detect. Thermodynamics is the physics of systems so large that their fluctuations are invisible, and statistical mechanics explains both the averages and the size of the fluctuations around them.

Counting with large numbers

Statistical mechanics deals in numbers so large that ordinary intuition fails. The number of ways to arrange the spins of a mole of magnetic atoms is \(2^{6 \times 10^{23}}\). Writing it out in decimal digits would take \(1.8 \times 10^{23}\) digits. Such numbers are best handled by their logarithms, and the most important tool for logarithms of factorials is Stirling's approximation,

It follows from writing \(\ln N! = \sum_{n=1}^{N}\ln n\) and approximating the sum by the integral \(\int_1^N \ln x\,dx = N\ln N - N + 1\); the last term, which matters only for small \(N\), comes from a more careful treatment. For the numbers met in thermal physics the first two terms are already extraordinarily accurate, and we will usually drop the last one. A related rule of thumb is that very large numbers behave strangely under multiplication: if \(\Omega = N^{N}\) with \(N = 10^{23}\), then multiplying \(\Omega\) by \(10^{23}\), or even by \(10^{1000}\), changes \(\ln\Omega\) by a negligible fraction. Only the exponent matters.

The second key fact is that probability distributions over macrostates become extremely sharp. If \(N\) independent spins each point up or down with probability one half, the number pointing up follows the binomial distribution, whose mean is \(N/2\) and whose standard deviation is \(\sqrt N/2\). The fraction pointing up therefore has mean \(1/2\) and standard deviation \(1/(2\sqrt N)\). For large \(N\) the binomial distribution is very close to a Gaussian, so the probability that the fraction differs from one half by more than \(\delta\) falls off roughly as \(\exp(-2N\delta^2)\). For a mole of spins, a deviation of one part in \(10^{10}\) has a probability of about \(10^{-5000}\). In practice, the most probable macrostate is the only one ever seen.

Fluctuations and Stirling's formula in code

The program below checks both facts. The first part compares Stirling's approximation with the exact \(\ln N!\), which Python computes with the log-gamma function. The second part flips \(N\) simulated coins 400 times and measures the spread of the fraction that land heads; the prediction is \(1/(2\sqrt N)\). The third part uses the Gaussian tail to estimate how unlikely macroscopic deviations are for a mole of spins.

import math, random
rnd = random.Random(361)

# 1. How good is Stirling's approximation?
print("Stirling's approximation for ln N!")
for N in (10, 100, 1000, 10**6):
    exact = math.lgamma(N + 1)
    crude = N * math.log(N) - N
    better = crude + 0.5 * math.log(2 * math.pi * N)
    print(f"  N = {N:>8}: ln N! = {exact:.6e}, "
          f"N ln N - N off by {abs(crude - exact) / exact:.1e}, "
          f"with sqrt(2 pi N) term off by {abs(better - exact) / exact:.1e}")

# 2. Fraction of N independent spins pointing up: average and scatter
print("\nFraction of spins up in N random spins (400 trials each)")
for N in (100, 1000, 10000):
    fr = []
    for _ in range(400):
        up = sum(1 for _ in range(N) if rnd.random() < 0.5)
        fr.append(up / N)
    m = sum(fr) / len(fr)
    sd = math.sqrt(sum((f - m) ** 2 for f in fr) / (len(fr) - 1))
    print(f"  N = {N:>6}: mean {m:.4f}, spread {sd:.5f}, "
          f"prediction 1/(2 sqrt N) = {0.5 / math.sqrt(N):.5f}")

# 3. Odds of a macroscopic deviation for a mole of spins
N = 6.022e23
for delta in (1e-10, 1e-11, 1e-12):
    # Gaussian tail: P ~ exp(-2 N delta^2) for fraction off by delta
    log10P = -2 * N * delta ** 2 / math.log(10)
    print(f"  mole of spins, fraction off by {delta:.0e}: "
          f"probability about 10^({log10P:.3g})")

Stirling's approximation for ln N!
  N =       10: ln N! = 1.510441e+01, N ln N - N off by 1.4e-01, with sqrt(2 pi N) term off by 5.5e-04
  N =      100: ln N! = 3.637394e+02, N ln N - N off by 8.9e-03, with sqrt(2 pi N) term off by 2.3e-06
  N =     1000: ln N! = 5.912128e+03, N ln N - N off by 7.4e-04, with sqrt(2 pi N) term off by 1.4e-08
  N =  1000000: ln N! = 1.281552e+07, N ln N - N off by 6.1e-07, with sqrt(2 pi N) term off by 6.5e-15

Fraction of spins up in N random spins (400 trials each)
  N =    100: mean 0.4988, spread 0.05268, prediction 1/(2 sqrt N) = 0.05000
  N =   1000: mean 0.5003, spread 0.01490, prediction 1/(2 sqrt N) = 0.01581
  N =  10000: mean 0.4997, spread 0.00510, prediction 1/(2 sqrt N) = 0.00500
  mole of spins, fraction off by 1e-10: probability about 10^(-5.23e+03)
  mole of spins, fraction off by 1e-11: probability about 10^(-52.3)
  mole of spins, fraction off by 1e-12: probability about 10^(-0.523)

Already at \(N = 100\) the crude form of Stirling's approximation is within one part in a hundred, and including \(\frac12\ln(2\pi N)\) makes it accurate to a few parts in a million. The simulated spreads agree with \(1/(2\sqrt N)\) to within the scatter expected from 400 trials. The last lines show the knife edge of large numbers. A mole of spins can easily differ from exactly half up by one part in \(10^{12}\), which is about one standard deviation; by one part in \(10^{11}\) the probability has fallen to \(10^{-52}\); by one part in \(10^{10}\), it is \(10^{-5000}\). Nothing in physics happens with a probability of \(10^{-5000}\).

The fundamental postulate and the arrow of time

Statistical mechanics rests on one assumption, the fundamental postulate: an isolated system in equilibrium is equally likely to be found in any of its accessible microstates. Accessible means consistent with the fixed total energy, volume and particle number. The postulate cannot be derived from mechanics alone. It is plausible because the dynamics of a large interacting system scrambles its microstate so thoroughly that, over time, it visits all the accessible states with equal frequency, an idea called the ergodic hypothesis. It is justified in the end by its success: every prediction derived from it, from the ideal gas law to the heat capacity of copper, agrees with experiment.

From the postulate, the second law follows as a statement of overwhelming probability. A macrostate with more microstates is more probable, in direct proportion to its multiplicity \(\Omega\). Left to itself, a system wanders through microstates and ends up, with near certainty, in the macrostate of largest \(\Omega\), which is the macrostate of maximum entropy \(S = k\ln\Omega\). The reverse process is not forbidden by the laws of motion, which are reversible, but it is so improbable that it never happens. A gas that fills a room will not spontaneously gather in one corner, not because any law prevents it, but because the corner macrostate has a fraction of roughly \(2^{-N}\) of the microstates of the spread-out one. This is how Ludwig Boltzmann, in papers from 1872 to 1877, reconciled the irreversibility of thermodynamics with the reversibility of mechanics.

Ensembles: three ways to set up the problem

Josiah Willard Gibbs, in his book Elementary Principles in Statistical Mechanics of 1902, introduced the idea of an ensemble: an imagined collection of many copies of a system, each in a different microstate, distributed according to the probabilities the postulate assigns. Averages over the ensemble stand in for averages over time. Three ensembles are used constantly, and this course develops each in turn:

For large systems all three give the same thermodynamics, because fluctuations in the quantities that are allowed to vary are negligible. The choice is one of convenience: each problem is easiest in one of them. A large part of the art of statistical mechanics is choosing the ensemble that makes the counting simple.

The plan of the course

Lessons 2 and 3 develop the microcanonical ensemble: counting microstates for spins, oscillators and the ideal gas, and deriving temperature, pressure and chemical potential from the entropy. Lesson 3 includes a full derivation of the Sackur–Tetrode formula for the entropy of an ideal gas, which PHYS-261 only stated. Lessons 4 to 6 develop the canonical ensemble and apply it to paramagnets, two-level systems, equipartition and the ideal gas of real molecules with their rotations and vibrations, including the surprising heat capacity of hydrogen. Lesson 7 introduces the grand canonical ensemble and the chemical potential. Lessons 8 to 10 treat quantum statistics, the Fermi–Dirac and Bose–Einstein distributions, and their applications: electrons in metals, white dwarf stars and the Chandrasekhar limit, blackbody radiation, phonons and Bose–Einstein condensation. Lesson 11 turns to interacting systems and phase transitions through the Ising model, mean-field theory, Landau theory and universality, building on the Monte Carlo simulation of the Ising model in the capstone of PHYS-234. Lesson 12 applies the whole toolkit to chemical equilibrium, adsorption, polymers and information.

The mathematics needed is calculus, some combinatorics and a few standard integrals, which are derived or listed where they are used. Every lesson includes a short Python program, using only the standard library, that computes the numbers quoted in the text.

A short history

The kinetic theory of gases, the first statistical physics, goes back to Daniel Bernoulli in 1738, but it became quantitative with James Clerk Maxwell's velocity distribution of 1860 and Boltzmann's work in the 1870s. Boltzmann's formula connecting entropy with the number of microstates, now written \(S = k\ln W\) and carved on his tombstone in Vienna, dates from 1877. Gibbs gave the subject its general form in 1902. Quantum mechanics transformed it: Max Planck's 1900 derivation of the blackbody spectrum was the first quantum calculation of all, Albert Einstein's 1907 theory of the heat capacity of solids was the first quantum theory of a material, and in 1924 to 1926 Satyendra Nath Bose, Einstein, Enrico Fermi and Paul Dirac found the two kinds of quantum statistics obeyed by identical particles. The exact solution of the two-dimensional Ising model by Lars Onsager in 1944 and the renormalization group of Kenneth Wilson around 1971, which earned the 1982 Nobel Prize, showed how statistical mechanics explains phase transitions. The subject continues to grow: Bose–Einstein condensation of a dilute atomic gas was achieved only in 1995.