Capstone: open questions — dark matter, dark energy, inflation and the Hubble tension

General relativity has passed every test it has been given, from the 43 arcseconds of Mercury to the ringing of merged black holes, and applied to the whole universe it yields a cosmological model that fits the microwave background, the abundances of light elements and the distribution of galaxies with a handful of parameters. Yet that model, \(\Lambda\)CDM, achieves its success by including two components that have never been detected in a laboratory and whose nature is unknown, and it rests on initial conditions that seem to require an explanation. This capstone draws the course together around the largest open questions: what dark matter is, what dark energy is, how the universe acquired its remarkable initial state, and whether the current disagreement between measurements of the expansion rate is a sign of new physics. In each case the tools are the ones developed in this course: geodesics, the field equations, the Friedmann equations and their tests.

The cosmic energy budget

The Planck results of lesson 11 imply that ordinary matter, everything made of atoms, is only about 5 parts in 100 of the energy density of the universe today. Dark matter accounts for about 27, and dark energy for about 68. Stars are a small fraction even of the ordinary matter; most baryons are in hot, diffuse gas between galaxies. Photons and neutrinos are a tiny remnant today, though they dominated the early universe.

Dark matter

The evidence for dark matter comes from gravity alone, and from many independent directions. Fritz Zwicky found in 1933 that galaxies in the Coma cluster move too fast to be held together by the visible mass. Vera Rubin and Kent Ford showed in the 1970s that the rotation curves of spiral galaxies stay flat far beyond the visible disc, as PHYS-430 described. Gravitational lensing, the light bending of lesson 7, maps the mass of clusters directly and finds about five times more than the visible galaxies and hot gas. In the Bullet Cluster, two clusters that collided about 150 million years ago, the hot gas, which carries most of the ordinary mass, was slowed by the collision, while the lensing mass passed through with the galaxies: the bulk of the mass does not interact like ordinary matter. And the microwave background's acoustic peaks of lesson 11 require a component that gravitates but does not feel radiation pressure, with \(\Omega_c h^2 = 0.120\), five times the baryon density, which is fixed independently by nucleosynthesis. Without dark matter, the small fluctuations of the microwave background would not have had time to grow into the galaxies we see.

What is it? The leading candidates are new particles. Weakly interacting massive particles, with masses from a few to thousands of times the proton mass, would naturally have the right abundance if they froze out of the early universe like the neutrons of lesson 11. Underground detectors, such as LZ in the Sanford laboratory in South Dakota, XENONnT in Italy and PandaX in China, use metric tons of liquid xenon to search for the rare recoils such particles would cause; they have found none, and have excluded a large part of the most natural parameter space. Axions, very light particles originally proposed to solve a puzzle in the strong interaction, are searched for with resonant microwave cavities in strong magnetic fields. Other ideas include sterile neutrinos and primordial black holes, the latter now constrained by microlensing surveys and gravitational-wave data across most mass ranges.

An alternative is that gravity itself differs from Einstein's theory on galactic scales. Modified Newtonian dynamics, proposed by Mordehai Milgrom in 1983, fits galaxy rotation curves remarkably well with a single acceleration scale of about \(1.2 \times 10^{-10}\) m/s². But it struggles with clusters, the Bullet Cluster, the microwave background and gravitational lensing, and relativistic versions of it were tightly constrained by GW170817's demonstration that gravitational waves and light travel together. Most cosmologists regard particle dark matter as the better explanation, though the question is open until the particle is found.

Dark energy and the cosmological constant

The acceleration of the expansion, discovered with supernovae in 1998 and confirmed by the microwave background and baryon acoustic oscillations, requires a component with negative pressure, \(w < -1/3\). The simplest candidate is Einstein's cosmological constant, \(w = -1\) exactly, equivalent to a constant energy density of empty space. Quantum field theory predicts that the vacuum has energy, but a naive estimate, summing zero-point energies of fields up to the Planck scale, gives a density about \(10^{120}\) times larger than observed, the worst prediction in physics. Even if supersymmetry or some other mechanism cancels most of it, explaining why the remainder is small but not zero, and comparable to the matter density today, is the cosmological constant problem. It has motivated ideas ranging from dynamical fields, called quintessence, whose energy slowly changes, to anthropic arguments in a multiverse.

Observations are now testing whether dark energy changes with time. The Dark Energy Spectroscopic Instrument, DESI, on the Mayall Telescope at Kitt Peak, Arizona, maps the three-dimensional positions of tens of millions of galaxies and quasars and measures the baryon acoustic scale at many redshifts. Its second data release in March 2025, combined with the microwave background and supernovae, showed a preference for dark energy whose density has been decreasing in recent billions of years, parametrized by \(w(a) = w_0 + w_a(1 - a)\) with \(w_0 > -1\) and \(w_a < 0\), at a significance of roughly three to four standard deviations depending on the supernova sample used. This is not yet the five standard deviations physicists require for a discovery, and the result depends on combining very different data sets. ESA's Euclid satellite, launched in 2023, and the Vera C. Rubin Observatory, which began its survey after releasing its first images in June 2025, together with NASA's Nancy Grace Roman Space Telescope, launched in 2026, will test it.

import math
c = 299792.458
def E(z, Om, w0=-1.0, wa=0.0, Or=9.1e-5):
    a = 1 / (1 + z)
    Ode = 1 - Om - Or
    # dark-energy density for w(a) = w0 + wa (1 - a)
    de = a ** (-3 * (1 + w0 + wa)) * math.exp(-3 * wa * (1 - a))
    return math.sqrt(Or * (1 + z)**4 + Om * (1 + z)**3 + Ode * de)
def dL(z, H0, Om, w0=-1.0, wa=0.0, n=2000):
    s = sum(1 / E((i + 0.5) * z / n, Om, w0, wa) for i in range(n)) * z / n
    return (1 + z) * c / H0 * s
def age(H0, Om, w0=-1.0, wa=0.0, n=20000):
    # t0 = integral over a of da / (a H); a from 0 to 1 with a = x^2
    tot = 0.0
    for i in range(n):
        x = (i + 0.5) / n; a = x * x; z = 1 / a - 1
        tot += 2 * x / (a * E(z, Om, w0, wa)) / n
    return tot / H0 * 3.0857e19 / 3.156e16
print("model                         age (Gyr)   d_L(z=1) (Mpc)   mu difference at z=1")
base = dL(1.0, 67.4, 0.315)
for name, w0, wa in (("cosmological constant", -1.0, 0.0), ("evolving, w0=-0.75 wa=-0.9", -0.75, -0.9), ("phantom, w0=-1.1", -1.1, 0.0)):
    d = dL(1.0, 67.4, 0.315, w0, wa)
    print(f"{name:28s}  {age(67.4, 0.315, w0, wa):8.2f}      {d:9.0f}        {5*math.log10(d/base):+.3f} mag")
# the cosmological constant problem: Planck energy density vs observed dark energy density
hbar, G = 1.0546e-34, 6.674e-11
c_ms = 2.998e8
rho_planck = c_ms**7 / (hbar * G**2)                       # J/m^3
rho_de = 0.685 * 3 * (67.4 / 3.0857e19)**2 / (8 * math.pi * G) * c_ms**2
print(f"Planck-scale vacuum energy {rho_planck:.1e} J/m^3, observed dark energy {rho_de:.1e} J/m^3, ratio 10^{math.log10(rho_planck/rho_de):.0f}")

model                         age (Gyr)   d_L(z=1) (Mpc)   mu difference at z=1
cosmological constant            13.79           6802        +0.000 mag
evolving, w0=-0.75 wa=-0.9       13.68           6685        -0.038 mag
phantom, w0=-1.1                 13.98           6927        +0.039 mag
Planck-scale vacuum energy 4.6e+113 J/m^3, observed dark energy 5.3e-10 J/m^3, ratio 10^123

Evolving dark energy of the kind DESI hints at changes distances to redshift 1 by only a few hundredths of a magnitude, which is why such exquisite surveys are needed to detect it. The cosmological constant problem, by contrast, is a mismatch of about 120 powers of ten. No accepted theory explains either.

Inflation

The hot Big Bang model leaves several features of the universe unexplained. The horizon problem: the microwave background has the same temperature to one part in 100,000 in every direction, yet in the standard model regions more than about two degrees apart on the sky were never in causal contact before last scattering. The flatness problem: the curvature term in the Friedmann equation grows relative to matter as the universe expands, so for space to be flat to half a part in a hundred today, it must have been flat to about one part in \(10^{62}\) at the Planck time. And the origin of the fluctuations: something had to produce the nearly scale-invariant, Gaussian density variations seen in the microwave background.

Inflation, proposed by Alan Guth in 1980 and developed by Andrei Linde, Andreas Albrecht, Paul Steinhardt, Alexei Starobinsky and others, addresses all three. It postulates a brief period very early on, perhaps \(10^{-36}\) s after the beginning, during which the universe was dominated by the potential energy of a scalar field, the inflaton, behaving like a temporary cosmological constant. The expansion was then exponential, \(a \propto e^{Ht}\), the de Sitter solution of lesson 10. In about \(10^{-32}\) s the universe grew by a factor of at least \(e^{60}\), about \(10^{26}\). A region small enough to have come to thermal equilibrium was stretched to encompass the entire observable universe, solving the horizon problem; any curvature was stretched flat, solving the flatness problem; and quantum fluctuations of the inflaton field were stretched to cosmic scales, becoming the seeds of structure. When inflation ended, the inflaton's energy decayed into particles, reheating the universe and starting the hot Big Bang.

Inflation makes testable predictions. The primordial fluctuations should be almost but not exactly scale invariant, with spectral index slightly less than one; Planck measured \(n_s = 0.965 \pm 0.004\), as predicted. Inflation should also have produced primordial gravitational waves, which would leave a distinctive swirling B-mode pattern in the polarization of the microwave background. Their amplitude, measured by the tensor-to-scalar ratio \(r\), depends on the energy scale of inflation. In 2014 the BICEP2 team announced a detection, but the signal turned out to be polarized emission from dust in our galaxy; the current upper limit, from BICEP/Keck and Planck, is \(r < 0.036\). Future experiments, including the Simons Observatory in Chile, aim to reach \(r \sim 0.001\). A detection would be direct evidence of quantum fluctuations of spacetime itself.

import math
# horizon problem: angle on today's sky subtended by the causal horizon at last scattering
c = 299792.458
H0, Om, Or = 67.4, 0.315, 9.1e-5
Ol = 1 - Om - Or
E = lambda z: math.sqrt(Or*(1+z)**4 + Om*(1+z)**3 + Ol)
def comoving(z1, z2, n=200000):
    # comoving distance between redshifts (Mpc), integrating in ln(1+z) for accuracy
    l1, l2 = math.log(1 + z1), math.log(1 + z2)
    tot = 0.0
    for i in range(n):
        x = l1 + (i + 0.5) * (l2 - l1) / n
        z = math.exp(x) - 1
        tot += (1 + z) / E(z) * (l2 - l1) / n
    return c / H0 * tot
z_ls = 1090
D_ls = comoving(0, z_ls)                 # distance to the last scattering surface
hor = comoving(z_ls, 1e9)                # comoving horizon at last scattering
print(f"distance to last scattering {D_ls:.0f} Mpc; horizon then {hor:.0f} Mpc (comoving)")
print(f"causally connected patches span about {math.degrees(2*hor/D_ls):.1f} degrees on the sky")
print(f"number of causally disconnected patches over the sky: about {4*math.pi/(math.pi*(hor/D_ls)**2):.0f}")
# flatness problem: |Omega - 1| grows as a^2 in radiation era, a in matter era
z_eq = 3400
today = 0.005
at_eq = today * (1 / (1 + z_eq))
T_eq_eV, T_planck_eV = 0.8, 1.22e28
at_planck = at_eq * (T_eq_eV / T_planck_eV) ** 2
print(f"|Omega - 1| needed: {at_eq:.1e} at matter-radiation equality, {at_planck:.0e} at the Planck time")
print(f"inflation by 60 e-folds stretches lengths by 10^{60/math.log(10):.0f} and suppresses curvature by 10^{120/math.log(10):.0f}")

distance to last scattering 13865 Mpc; horizon then 281 Mpc (comoving)
causally connected patches span about 2.3 degrees on the sky
number of causally disconnected patches over the sky: about 9744
|Omega - 1| needed: 1.5e-06 at matter-radiation equality, 6e-63 at the Planck time
inflation by 60 e-folds stretches lengths by 10^26 and suppresses curvature by 10^52

The Hubble tension

The expansion rate today can be measured in two very different ways. The early-universe method, lesson 11's, fits the \(\Lambda\)CDM model to the microwave background and infers \(H_0 = 67.4 \pm 0.5\) km/s/Mpc; baryon acoustic oscillations combined with nucleosynthesis give a consistent value. The late-universe method, the distance ladder of PHYS-430, calibrates Cepheid variables with parallaxes, uses them to calibrate Type Ia supernovae in nearby galaxies, and then measures supernovae in the Hubble flow. The SH0ES team led by Adam Riess finds \(H_0 = 73.0 \pm 1.0\) km/s/Mpc. The difference is about five standard deviations. JWST observations have confirmed the Cepheid photometry, ruling out crowding by nearby stars as the cause. An independent ladder using the tip of the red giant branch, by the CCHP team led by Wendy Freedman, gives about 70.4 km/s/Mpc, between the two, and the debate over systematic errors in the ladders is active.

If the tension is real, \(\Lambda\)CDM is incomplete. Proposed explanations include early dark energy that briefly increased the expansion rate before recombination, shrinking the sound horizon so that the same microwave background angle implies a larger \(H_0\); extra relativistic species; interactions between dark matter and dark energy; or evolving dark energy of the kind DESI hints at, though that tends to move \(H_0\) in the wrong direction. None is fully satisfactory. Independent methods are the way forward. Gravitational-wave standard sirens, from lesson 9, measure distances without any ladder; with a few dozen events with identified host galaxies, they could reach the precision needed to decide. Time delays between the multiple images of lensed quasars, an application of lesson 7's Shapiro delay on a cosmic scale, give another independent route.

import math
measurements = [("Planck CMB (LCDM)", 67.4, 0.5), ("SH0ES Cepheids + SNe Ia", 73.0, 1.0),
                ("CCHP red giant tip", 70.39, 1.9), ("GW170817 standard siren", 70.0, 10.0)]
for name, h, s in measurements:
    print(f"{name:26s} {h:6.2f} +/- {s:4.1f} km/s/Mpc   age if LCDM ~ {13.80*67.4/h:5.2f} Gyr")
(_, a, sa), (_, b, sb) = measurements[0], measurements[1]
print(f"Planck vs SH0ES: difference {b-a:.1f} km/s/Mpc = {(b-a)/math.hypot(sa, sb):.1f} standard deviations")
# how many standard sirens to reach 2 km/s/Mpc? each well-localized event gives roughly 0.15 fractional error
per_event = 0.15 * 70
for n in (1, 10, 50, 100):
    print(f"{n:4d} sirens with host galaxies: uncertainty about {per_event/math.sqrt(n):5.1f} km/s/Mpc")

Planck CMB (LCDM)           67.40 +/-  0.5 km/s/Mpc   age if LCDM ~ 13.80 Gyr
SH0ES Cepheids + SNe Ia     73.00 +/-  1.0 km/s/Mpc   age if LCDM ~ 12.74 Gyr
CCHP red giant tip          70.39 +/-  1.9 km/s/Mpc   age if LCDM ~ 13.21 Gyr
GW170817 standard siren     70.00 +/- 10.0 km/s/Mpc   age if LCDM ~ 13.29 Gyr
Planck vs SH0ES: difference 5.6 km/s/Mpc = 5.0 standard deviations
   1 sirens with host galaxies: uncertainty about  10.5 km/s/Mpc
  10 sirens with host galaxies: uncertainty about   3.3 km/s/Mpc
  50 sirens with host galaxies: uncertainty about   1.5 km/s/Mpc
 100 sirens with host galaxies: uncertainty about   1.1 km/s/Mpc

General relativity meets quantum mechanics

The deepest open question underlies all the others. General relativity is a classical theory; the rest of physics is quantum. At the Planck scale, lengths of \(1.6 \times 10^{-35}\) m and energies of \(1.2 \times 10^{19}\) GeV, quantum fluctuations of spacetime itself should become important, and general relativity's predictions of singularities, at the Big Bang and inside black holes, signal its breakdown. String theory and loop quantum gravity are the leading candidates for a quantum theory of gravity, and the holographic principle, the black-hole entropy of lesson 8 and the information paradox are the main clues, but no experiment has yet probed quantum gravity directly. Hawking radiation, primordial gravitational waves from inflation, and precision tests of gravity at short distances and in the strongest fields are the windows through which it might first be seen.

It is worth closing where the course began. In 1907 Einstein noticed that a falling person feels no weight. From that observation, a demand for mathematical consistency and an equation fixed by Newton's limit came a theory that predicted black holes, gravitational waves and an expanding universe, phenomena that took most of a century to observe and that are now the daily work of astronomers. The open questions of this lesson are open precisely because the theory is so well tested that its failures, if they come, will be unmistakable.