Why thermal physics: from steam engines to the arrow of time

Thermal physics is the study of systems with enormous numbers of parts: the \(10^{25}\) molecules of air in a room, the vibrating atoms of a metal bar, the photons inside a furnace, the electrons in a star. We cannot follow each part, and we would not want to. Instead we describe such systems by a handful of quantities, such as temperature, pressure, volume and energy, and we look for laws that connect them. Those laws turn out to be among the most general in all of science. They govern steam turbines and refrigerators, the boiling of water and the freezing of lakes, the heat capacity of solids, the glow of the Sun, the climate of the Earth and even the minimum energy needed to erase a bit of computer memory.

This course has two strands that are woven together. The first is thermodynamics: a set of laws about heat, work and temperature that were discovered in the nineteenth century by engineers and physicists trying to understand engines, long before anyone was sure that atoms existed. Its power comes from the fact that it does not depend on any model of what matter is made of. The second strand is statistical mechanics: the explanation of those laws in terms of the behavior of atoms and molecules, using probability. Thermodynamics tells us that entropy never decreases in an isolated system; statistical mechanics tells us why, and how improbable a decrease would be. This first lesson surveys the ground, introduces the central ideas informally and shows a small simulation in which the arrow of time appears on its own.

A short history

People have used fire and steam for millennia, but the science of heat began in earnest with thermometers. Daniel Gabriel Fahrenheit made reliable mercury thermometers and proposed his scale in 1724, and Anders Celsius proposed his centigrade scale in 1742. In the 1760s Joseph Black in Scotland distinguished temperature from quantity of heat: he showed that melting ice absorbs heat without getting warmer, the latent heat, and that different substances need different amounts of heat to warm by the same amount, the heat capacity. Black thought of heat as a weightless fluid, caloric, that flowed from hot bodies to cold ones. The idea was useful, and many correct results were derived with it.

James Watt, who worked as an instrument maker at Black's university, made the steam engine efficient by adding a separate condenser, patented in 1769, and the engines built by his partnership with Matthew Boulton from 1776 onward drove the Industrial Revolution. Engines raised a scientific question that nobody could yet answer: how much work can be obtained from a given amount of fuel? In 1798 Benjamin Thompson, Count Rumford, observed that boring cannon produced apparently unlimited heat from friction, which was hard to reconcile with a conserved fluid. In 1824 a young French engineer, Sadi Carnot, published a short book, Reflections on the Motive Power of Fire, which showed that the efficiency of an ideal engine depends only on the temperatures between which it works. Carnot still used caloric, yet his conclusion survived when caloric did not.

In the 1840s James Prescott Joule measured the mechanical equivalent of heat with falling weights that stirred water, showing that work and heat are interchangeable forms of energy. Hermann von Helmholtz and others stated the conservation of energy in general form, now the first law of thermodynamics. Rudolf Clausius in 1850 and William Thomson, later Lord Kelvin, in 1851 reconciled Carnot's theorem with energy conservation, giving the second law. Kelvin had already proposed an absolute temperature scale in 1848, based on Carnot's work rather than on any particular substance. In 1865 Clausius introduced the word entropy and summarized the subject in two sentences: the energy of the universe is constant, and the entropy of the universe tends to a maximum.

The statistical strand began with James Clerk Maxwell, who in 1860 derived the distribution of molecular speeds in a gas. Ludwig Boltzmann extended Maxwell's work, proved in 1872 that collisions drive any gas toward Maxwell's distribution, and in 1877 connected entropy to the number of microscopic arrangements, the relation now written \(S = k \ln \Omega\) and carved on his tombstone in Vienna. Josiah Willard Gibbs at Yale wrote the definitive treatment of chemical thermodynamics in 1876 to 1878 and later founded the ensemble method of statistical mechanics. In 1900 Max Planck explained the spectrum of thermal radiation by assuming that energy is exchanged in quanta, and in 1907 Albert Einstein used the same idea to explain why the heat capacity of solids falls at low temperatures. Thermal physics was thus one of the two doorways to quantum mechanics.

Macrostates and microstates

The central distinction of the subject is between a macrostate and a microstate. A macrostate is what we measure: for a gas, its pressure, volume, temperature and amount. A microstate is a complete microscopic description: the position and velocity of every molecule, or, in quantum terms, which quantum state the whole system occupies. An enormous number of microstates correspond to the same macrostate. Molecules move and collide all the time, so the microstate changes at every instant, while the macrostate stays the same once the system has settled down.

That settled condition is thermal equilibrium: the state in which the macroscopic quantities no longer change with time. A cup of coffee left on a desk cools until it reaches the temperature of the room, then stays there. A gas released into one corner of a box spreads until it fills the box uniformly. We will define temperature carefully in lesson 2, but its essential property is already visible: two systems in thermal contact exchange energy until their temperatures are equal, and then the net exchange stops.

The statistical explanation of equilibrium is simple to state. If every microstate compatible with the constraints is equally likely, which is the fundamental assumption of statistical mechanics, then the system will spend almost all of its time in the macrostate that has the most microstates. For large systems this most probable macrostate is not just slightly more probable than the others; it is overwhelmingly more probable. The approach to equilibrium is a drift from macrostates with few microstates to macrostates with many, and the reverse drift is not forbidden, only absurdly unlikely.

Counting gives the arrow of time

Consider \(N\) molecules in a box divided into two equal halves. Each molecule is equally likely to be found in either half, independently of the others, so the probability that all \(N\) are in the left half at a given moment is \(2^{-N}\). For 10 molecules that is about one in a thousand, an event you would see if you watched for a while. For 100 molecules it is \(10^{-30}\). For a mole of gas it is \(10^{-1.8 \times 10^{23}}\), a number so small that the age of the universe measured in the shortest conceivable time intervals would not make a dent in it. This is why the air in a room never collects in one corner, even though no law of mechanics forbids it.

The number of ways to have exactly \(n\) of the \(N\) molecules on the left is the binomial coefficient

which is largest at \(n = N/2\). The distribution of \(n\) has mean \(N/2\) and standard deviation \(\sqrt{N}/2\), so the fraction on the left fluctuates by about \(1/(2\sqrt{N})\). For \(N = 10^{24}\) the fluctuation is one part in \(10^{12}\), far below anything we could measure. Thermodynamic quantities are sharp not because the microscopic world is orderly but because of the law of large numbers.

Paul and Tatyana Ehrenfest proposed a simple model of this behavior in 1907. Put \(N\) numbered balls in two urns. At each step, choose a number at random and move that ball to the other urn. If the left urn holds \(n\) balls, the next move takes a ball from left to right with probability \(n/N\) and from right to left with probability \((N - n)/N\). The rule is perfectly symmetric in time: any sequence of moves is exactly as likely as the same sequence run backwards. Yet when we start with every ball on the left, the number on the left falls steadily toward \(N/2\) and then fluctuates around it. The average excess over \(N/2\) shrinks by a factor \(1 - 2/N\) at each step, so after about \(N\) steps it has fallen to \(e^{-2}\), about 14% of its starting value.

import math, random

# 1. Chance that all N molecules are found in the left half of a box
for N in (10, 100, 1000):
    print(f"N = {N:5d}: probability all in left half = 2^-{N} = 10^{-N * math.log10(2):.1f}")
NA = 6.02214076e23
print(f"N = one mole: log10(probability) = {-NA * math.log10(2):.3e}")

# 2. Ehrenfest urn model: N balls in two urns, move a random ball each step
def ehrenfest(N, steps, seed=261):
    rnd = random.Random(seed)
    left = N                    # start with every ball in the left urn
    history = []
    for t in range(steps + 1):
        if t % (steps // 10) == 0:
            history.append((t, left))
        if rnd.randrange(N) < left:
            left -= 1
        else:
            left += 1
    return history

for N in (20, 2000):
    print(f"\nEhrenfest model with N = {N}:")
    for t, left in ehrenfest(N, 10 * N):
        bar = "#" * round(40 * left / N)
        print(f"  step {t:6d}  left fraction {left / N:5.3f}  {bar}")

# 3. Size of equilibrium fluctuations: standard deviation of the left fraction
for N in (20, 2000, 200000):
    print(f"N = {N:6d}: fluctuation of left fraction ~ 1/(2 sqrt N) = {0.5 / math.sqrt(N):.5f}")

N =    10: probability all in left half = 2^-10 = 10^-3.0
N =   100: probability all in left half = 2^-100 = 10^-30.1
N =  1000: probability all in left half = 2^-1000 = 10^-301.0
N = one mole: log10(probability) = -1.813e+23

Ehrenfest model with N = 20:
  step      0  left fraction 1.000  ########################################
  step     20  left fraction 0.500  ####################
  step     40  left fraction 0.500  ####################
  step     60  left fraction 0.500  ####################
  step     80  left fraction 0.600  ########################
  step    100  left fraction 0.700  ############################
  step    120  left fraction 0.500  ####################
  step    140  left fraction 0.700  ############################
  step    160  left fraction 0.400  ################
  step    180  left fraction 0.400  ################
  step    200  left fraction 0.400  ################

Ehrenfest model with N = 2000:
  step      0  left fraction 1.000  ########################################
  step   2000  left fraction 0.568  #######################
  step   4000  left fraction 0.499  ####################
  step   6000  left fraction 0.506  ####################
  step   8000  left fraction 0.498  ####################
  step  10000  left fraction 0.510  ####################
  step  12000  left fraction 0.498  ####################
  step  14000  left fraction 0.501  ####################
  step  16000  left fraction 0.486  ###################
  step  18000  left fraction 0.494  ####################
  step  20000  left fraction 0.488  ####################
N =     20: fluctuation of left fraction ~ 1/(2 sqrt N) = 0.11180
N =   2000: fluctuation of left fraction ~ 1/(2 sqrt N) = 0.01118
N = 200000: fluctuation of left fraction ~ 1/(2 sqrt N) = 0.00112

With 20 balls the left fraction reaches one half within a few dozen steps and then wanders between 0.4 and 0.7; a return to all 20 on the left would happen eventually, about once in \(2^{20}\), a million, steps on average. With 2000 balls the fraction after 2000 steps is 0.568, exactly as the factor \(e^{-2}\) predicts, and from then on it stays within a few hundredths of one half. The fluctuations shrink like \(1/\sqrt{N}\). Nothing in the rule distinguishes past from future; the arrow of time comes entirely from starting in a very special, low-probability state and letting chance do the rest. The same is true of the universe as a whole, which appears to have begun in a state of extraordinarily low entropy, a puzzle that remains open in cosmology.

The laws of thermodynamics in brief

The laws are traditionally numbered from zero to three. The zeroth law says that if two systems are each in thermal equilibrium with a third, they are in equilibrium with each other; it is what makes thermometers possible and temperature meaningful, and it was named last, in the 1930s, after the others were already numbered. The first law says that energy is conserved: the change in a system's internal energy equals the heat added to it minus the work it does. The second law says that the entropy of an isolated system never decreases; equivalent statements are that heat does not flow spontaneously from cold to hot and that no engine can turn heat completely into work in a cycle. The third law, formulated by Walther Nernst in 1906, says that entropy approaches a constant, which can be taken as zero for a perfect crystal, as the temperature approaches absolute zero, so that absolute zero cannot be reached in a finite number of steps.

We will meet each of these laws in turn: temperature and the zeroth law in lesson 2, the first law in lesson 3, the second law and its consequences for engines and heat pumps in lessons 7 and 8, and the third law when we compute heat capacities and entropies near absolute zero. Lessons 10 and 11 then explain the laws statistically, and the capstone applies them to the climate, the stars and the physics of information.

Thermal physics and computation

Throughout the course we will use short Python programs, written with the standard library only, to make the ideas concrete. Several of them are simulations in the spirit of the Ehrenfest model: we follow a few thousand particles exchanging energy by random rules and watch thermodynamic behavior emerge. Others evaluate the formulas we derive for realistic cases: the heat loss through a wall on a cold Minneapolis night, the efficiency of a heat pump, the temperature of the Earth. If you have taken the computational physics course, you will recognize the Metropolis algorithm and the Ising model from its capstone; here we explain why they work. The programs are short enough to read in full, and you are encouraged to change their parameters and see what happens.

A good habit is to ask of every result whether it makes sense in orders of magnitude. Room temperature is about 300 K, and the thermal energy per particle \(kT\) at that temperature is about \(4 \times 10^{-21}\) J, or \(\frac{1}{40}\) eV, where \(k = 1.380649 \times 10^{-23}\) J/K is the Boltzmann constant. Chemical bonds are a few eV, which is why most molecules survive at room temperature. A mole contains \(N_A = 6.022 \times 10^{23}\) particles, and \(N_A k = R = 8.314\) J/(mol K) is the gas constant. These few numbers, used carefully, let you estimate a remarkable range of everyday phenomena.