BIOL-321 · Lesson 7 of 12 · 28 min

Species and speciation

Darwin called his book On the Origin of Species, yet he said relatively little about how one species splits into two. Today speciation is one of the most active areas of evolutionary biology, combining field studies of hybrid zones, laboratory crosses, genome sequencing and mathematical models. This lesson asks what a species is, how barriers to gene flow arise, where and how quickly speciation happens and what genes are involved. BIOL-108 introduced speciation; here we go deeper into the genetics and geography of the process.

What is a species?

There is no single definition of species that works for every organism, and biologists have proposed more than twenty. The most widely used is the biological species concept, formulated by Ernst Mayr in 1942: species are groups of interbreeding natural populations that are reproductively isolated from other such groups. Its strength is that it identifies the process, reproductive isolation, that keeps species distinct. Its weaknesses are that it cannot be applied to asexual organisms or fossils, and that it is hard to test for populations that do not meet in nature.

Some species concepts and their trade-offs
ConceptDefinition in briefStrengthsLimitations
BiologicalInterbreeding populations reproductively isolated from othersFocuses on gene flow, the process that maintains speciesNot applicable to asexual organisms or fossils; hybridization blurs lines
MorphologicalGroups distinguished by consistent physical differencesUsable for fossils and museum specimensCryptic species look alike; one species may be very variable
PhylogeneticSmallest group of organisms sharing a common ancestor, diagnosable by unique traitsApplies to all organisms; uses genetic dataCan split groups very finely, inflating species counts
EcologicalA lineage occupying a distinct nicheLinks species to adaptationNiches are hard to define and can overlap
General lineageSeparately evolving metapopulation lineagesUnifies the others as different kinds of evidenceDoes not by itself say when a lineage is separate enough

Kevin de Queiroz proposed in 1998 that most concepts agree that species are separately evolving lineages, and differ only in which property, such as reproductive isolation, diagnosability or niche difference, they treat as the threshold. Because these properties arise at different times during divergence, disagreements about whether two populations are distinct species often reflect the fact that speciation is a gradual process caught partway through.

Cryptic species are common. Genetic barcoding of the skipper butterfly Astraptes fulgerator in Costa Rica, published in 2004, revealed that what had been considered one species was at least ten, with caterpillars feeding on different plants. In bacteria and archaea, where genes move between lineages by horizontal transfer, species are usually defined operationally, for example by about 95 % average nucleotide identity across the genome.

Reproductive isolating barriers

Reproductive isolation is produced by barriers that prevent interbreeding or reduce the fitness of hybrids. Prezygotic barriers act before fertilization; postzygotic barriers act after.

Reproductive isolating barriers
StageBarrierExample
PrezygoticHabitat isolationApple and hawthorn races of Rhagoletis flies mate on their host fruit
PrezygoticTemporal isolationPeriodical cicada broods emerge in different years; corals spawn on different nights
PrezygoticBehavioral isolationFemale frogs respond only to the call of their own species
PrezygoticMechanical isolationGenital shape differs between damselfly species; flowers fit different pollinators
PrezygoticGametic isolationSea urchin sperm protein bindin binds only eggs of its own species
PostzygoticHybrid inviabilityMany interspecific frog hybrids fail to develop
PostzygoticHybrid sterilityMules (horse × donkey) are almost always sterile
PostzygoticHybrid breakdownSecond-generation hybrids of cultivated rice varieties are weak or sterile

Total isolation is the combined effect of barriers acting in sequence. Following the method of Douglas Schemske and colleagues, if each barrier i reduces gene flow by a fraction RI_i of what remains after the previous barriers, the total isolation is

\[RI_{total} = 1 - \prod_{i}(1 - RI_i)\]
Combining sequential barriers into total reproductive isolation (monkeyflower values rounded)
import numpy as np
# Sequential reproductive isolation between two monkeyflower species (values in the spirit of
# Ramsey, Bradshaw and Schemske 2003 for Mimulus lewisii and M. cardinalis; rounded for teaching)
barriers = {"ecogeographic": 0.587, "pollinator": 0.976, "pollen competition": 0.62,
            "hybrid seed set": 0.0, "hybrid fitness": 0.0}
remaining = 1.0
for name, ri in barriers.items():
    contrib = remaining * ri
    remaining -= contrib
    print(f"{name:20s} RI = {ri:5.3f}  absolute contribution = {contrib:6.4f}  gene flow left = {remaining:.5f}")
print(f"total isolation = {1 - remaining:.5f}")
Output
ecogeographic        RI = 0.587  absolute contribution = 0.5870  gene flow left = 0.41300
pollinator           RI = 0.976  absolute contribution = 0.4031  gene flow left = 0.00991
pollen competition   RI = 0.620  absolute contribution = 0.0061  gene flow left = 0.00377
hybrid seed set      RI = 0.000  absolute contribution = 0.0000  gene flow left = 0.00377
hybrid fitness       RI = 0.000  absolute contribution = 0.0000  gene flow left = 0.00377
total isolation = 0.99623

Barriers acting early in the life cycle contribute most to total isolation, because later barriers only act on whatever gene flow remains. In the monkeyflowers, differences in where the two species grow and in pollinator preference (bumblebees visit pink-flowered Mimulus lewisii and hummingbirds visit red M. cardinalis) together account for more than 99 % of total isolation, even though hybrids are fertile in the greenhouse. Pollinator preference has the highest individual RI, but because it acts after ecogeographic isolation, its absolute contribution is smaller. Prezygotic isolation is often strong in young species pairs, and comparative studies of Drosophila by Jerry Coyne and H. Allen Orr in 1989 and 1997 found that prezygotic isolation evolved faster in species pairs whose ranges overlap, as expected if natural selection strengthens it.

Dobzhansky–Muller incompatibilities

How can hybrid sterility evolve, given that any intermediate step that reduced fertility would be selected against? Theodosius Dobzhansky (1937) and Hermann Muller (1942) solved the puzzle. Suppose an ancestral population has genotype aabb. In one isolated population, a new allele A arises and spreads; AAbb is perfectly fit. In the other, B arises and spreads; aaBB is fit. A and B have never been together in one genome. When the populations meet, hybrids carry A and B together, and if the two alleles interact badly, the hybrid suffers. No population ever passed through a less fit state.

A diagram of an ancestral population with genotype aabb splitting into two lineages. The left lineage becomes AAbb and the right becomes aaBB, each marked as fully fit. The two lineages then meet to form a hybrid AaBb, in which A and B interact for the first time, marked as reduced fitness.
The Dobzhansky–Muller model: incompatible alleles evolve in separate populations and meet only in hybrids

Orr showed in 1995 that the number of incompatibilities should increase faster than linearly with divergence time, a pattern called the snowball effect, because each new substitution can interact with all the previous substitutions in the other lineage. If K substitutions separate the lineages, the number of potential pairwise incompatibilities grows roughly with K squared.

The snowball effect: incompatibilities accumulate faster than substitutions
import numpy as np
rng = np.random.default_rng(321)
# Snowball effect: each new substitution can be incompatible with every substitution already in the other lineage
p = 0.002                      # chance a given cross-lineage pair is incompatible (illustrative)
for K in (10, 20, 40, 80):
    k1 = K // 2; k2 = K - k1
    pairs = k1 * k2            # cross-lineage pairs that never met before
    sims = rng.binomial(pairs, p, 5000)
    print(f"K = {K:3d} substitutions: expected DMIs = {p*pairs:5.2f}, simulated mean = {sims.mean():5.2f}, "
          f"chance hybrid has at least one = {np.mean(sims > 0):.2f}")
Output
K =  10 substitutions: expected DMIs =  0.05, simulated mean =  0.05, chance hybrid has at least one = 0.05
K =  20 substitutions: expected DMIs =  0.20, simulated mean =  0.19, chance hybrid has at least one = 0.18
K =  40 substitutions: expected DMIs =  0.80, simulated mean =  0.82, chance hybrid has at least one = 0.56
K =  80 substitutions: expected DMIs =  3.20, simulated mean =  3.20, chance hybrid has at least one = 0.96

Doubling the number of substitutions roughly quadruples the expected number of incompatibilities. Experiments in Drosophila and in tomatoes, where Leonie Moyle and Takuya Nakazato introgressed segments of one species into another in 2010, have found evidence for the snowball. Several genes causing hybrid incompatibility have been identified, such as Hmr and Lhr in Drosophila, which interact to kill hybrid males, and Prdm9 in mice, a gene that directs where recombination occurs and causes hybrid male sterility when copies from different subspecies are combined.

Two regular patterns describe hybrid problems. Haldane’s rule (1922) states that when only one sex of hybrids is sterile or inviable, it is usually the heterogametic sex: XY males in mammals and flies, ZW females in birds and butterflies. A leading explanation is dominance theory: recessive incompatibilities on the X (or Z) chromosome are exposed in the heterogametic sex, which has only one copy. The large X effect is the observation that the X chromosome contributes disproportionately to hybrid incompatibility.

The geography of speciation

Speciation is classified by the degree of gene flow during divergence, often described geographically.

Geographic modes of speciation
ModeGeographyGene flow during divergenceExample
Allopatric (vicariance)Populations separated by a new barrierNoneSnapping shrimp species pairs on either side of the Isthmus of Panama, closed about 3 million years ago
PeripatricA small population founds a new colonyNone after foundingHawaiian Drosophila on successively younger islands
ParapatricAdjacent populations along an environmental gradientLimited, across a boundarySweet vernal grass on and off mine tailings with heavy metals
SympatricSame areaPotentially highCichlid fish in crater lakes; apple maggot fly host races; polyploid plants

Allopatric speciation is thought to be the most common mode, because geographic separation removes the homogenizing effect of gene flow and allows populations to diverge by selection and drift. The Isthmus of Panama provides a natural experiment: Nancy Knowlton and colleagues found in 1993 that snapping shrimp pairs separated by the isthmus show substantial reproductive isolation, with older pairs, likely separated before final closure, more isolated than younger ones.

Sympatric speciation, divergence without geographic separation, was long doubted, because gene flow and recombination break up associations between traits needed for adaptation and those needed for mate choice. It is now accepted that it can happen under particular conditions, especially when a single trait affects both ecology and mating (a so-called magic trait), or when mating occurs on the host. Midas cichlids in the Nicaraguan crater Lake Apoyo and the cichlids of the tiny crater Lake Barombi Mbo in Cameroon, where 11 endemic species evolved, are leading candidates. The clearest form of sympatric speciation is polyploidy, the doubling of chromosome sets, which creates instant reproductive isolation because crosses between a tetraploid and its diploid parent produce triploids that are largely sterile. A substantial fraction of flowering plant speciation events involve polyploidy, estimated at about 15 % in a 2009 study by Troy Wood and colleagues. Bread wheat is a hexaploid formed by two rounds of hybridization and polyploidy.

Ring species and hybrid zones

Ring species show speciation stretched out in space. The classic example is the greenish warbler, which circles the Tibetan Plateau. Neighboring populations interbreed around the ring, but where the two ends meet in Siberia, they differ in song and do not interbreed, although Darren Irwin’s genetic work in 2005 found that the ring has had breaks in its history. The Ensatina salamanders of California are another example. Ring species show that the small differences between populations and the large differences between species lie on a continuum.

Hybrid zones are regions where genetically distinct populations meet and produce hybrids. Many are tension zones, maintained by a balance between dispersal into the zone and selection against hybrids. In a cline model, the width w of the zone scales with dispersal distance σ divided by the square root of the strength of selection s against hybrids:

\[w \approx \frac{\sigma}{\sqrt{s}}\]

Worked problem

Using cline width to estimate selection

Across a hybrid zone between two grasshopper subspecies, a diagnostic allele changes from 0.05 to 0.95 frequency over a width of about 2 km. Individuals disperse a typical distance σ of about 400 m per generation. Using the approximation w = σ/√s (with a constant of order one), estimate s.

Show solution
  1. Rearrange: √s = σ/w = 0.4 km / 2 km = 0.2.
  2. s = 0.2² = 0.04.
  3. Selection against hybrids of about 4 % per locus is enough to maintain a narrow zone against this level of dispersal.

Answer: s ≈ 0.04, modest selection that nonetheless keeps the subspecies distinct.

Hybrid zones are natural laboratories. Some genes flow across freely, while others are held back by selection; genome scans reveal islands of divergence, regions with high differentiation among a background of shared variation. In the hybrid zone between carrion and hooded crows in Europe, the populations are nearly identical across the genome except for a small region controlling plumage color, maintained by mate preferences. Hybridization can also create new species, called hybrid speciation, as in Helianthus sunflowers adapted to sand dunes and salt marshes and in the Italian sparrow, a stable hybrid of house and Spanish sparrows.

If hybrids are less fit, selection may favor stronger prezygotic isolation where species meet, a process called reinforcement. In the European pied and collared flycatchers, females in areas of overlap are choosier and males differ more in plumage than in allopatry, a pattern called reproductive character displacement.

How fast does speciation happen?

Speciation can take anywhere from a few generations to millions of years. Polyploid plants can form in a single generation. The cichlids of Lake Victoria, which dried out completely about 15,000 years ago according to sediment cores, now include around 500 species, indicating extraordinarily rapid speciation. For most animals, the median time to complete reproductive isolation appears to be on the order of one to a few million years. Rates differ among lineages for many reasons, including sexual selection, ecological opportunity and generation time.

Speciation rates can be estimated from phylogenies (Lesson 9). Under a simple model, if a clade grows from one lineage to n lineages in time t with speciation rate λ and extinction rate μ, the net diversification rate r = λ − μ is approximately

\[r \approx \frac{\ln n}{t}\]

For example, if Hawaiian silverswords, about 30 species, descended from a single ancestor that arrived around 5 million years ago, r is approximately ln 30/5 ≈ 0.68 per million years, a high rate compared with typical plant lineages. This estimate ignores extinction and assumes steady growth, so it is a rough guide.

Common mistakes

  • MistakeThere is one correct definition of species.

    InsteadDifferent concepts emphasize different properties of separately evolving lineages; each has uses and limitations.

  • MistakeHybrid sterility must evolve through a less fit intermediate.

    InsteadDobzhansky–Muller incompatibilities arise because alleles that are fine in their own population interact badly only in hybrids.

  • MistakeTotal isolation is the sum of barrier strengths.

    InsteadBarriers act in sequence, so later barriers act only on the gene flow left after earlier ones.

  • MistakeSympatric speciation is impossible.

    InsteadIt is rarer, but polyploidy and cases with magic traits or host-associated mating show it can occur.

  • MistakeHybridization is always an evolutionary dead end.

    InsteadHybridization can transfer adaptive genes and can create new species.

  • MistakeRing species are perfect snapshots of speciation in progress.

    InsteadThey illustrate the continuum, but some rings, such as the greenish warbler, have had breaks in their history.

Summary and key terms

  • Species concepts differ in which property of separately evolving lineages they treat as decisive; the biological species concept emphasizes reproductive isolation.
  • Prezygotic and postzygotic barriers combine sequentially, and early barriers usually contribute most to total isolation.
  • Dobzhansky–Muller incompatibilities explain how hybrid problems evolve without fitness valleys, and they accumulate faster than linearly with divergence.
  • Haldane’s rule and the large X effect describe regular patterns in hybrid sterility and inviability.
  • Allopatric speciation is probably most common; sympatric speciation occurs, especially through polyploidy.
  • Hybrid zones and ring species reveal speciation as a continuum, and hybridization can sometimes create new species.
  • Speciation can take one generation (polyploidy), thousands of years (cichlids) or millions of years.
biological species concept
Species are groups of interbreeding populations reproductively isolated from others.
cryptic species
Species that look alike but are genetically and reproductively distinct.
prezygotic barrier
A reproductive barrier acting before fertilization.
postzygotic barrier
A reproductive barrier reducing hybrid viability or fertility.
Dobzhansky–Muller incompatibility
A harmful interaction between alleles that evolved separately and meet in hybrids.
Haldane’s rule
If one hybrid sex is sterile or inviable, it is usually the heterogametic sex.
allopatric speciation
Speciation in geographically separated populations.
polyploidy
Possession of more than two complete sets of chromosomes.
hybrid zone
A region where distinct populations meet and interbreed.
reinforcement
Selection for stronger prezygotic isolation where hybrids are less fit.

Practice problems

Work each problem on paper first, then reveal the answer to check.

  1. Why can the biological species concept not be applied to fossils?

    Show answer

    Interbreeding cannot be observed or tested in fossils.

  2. Two barriers have RI = 0.8 and 0.5. What is total isolation?

    Show answer

    1 − (0.2 × 0.5) = 0.9.

  3. Give an example of gametic isolation.

    Show answer

    Sea urchin bindin proteins allow sperm to bind only eggs of their own species.

  4. Explain the Dobzhansky–Muller model in one sentence.

    Show answer

    Alleles that are harmless in their own populations interact badly when combined in hybrids.

  5. What is the snowball effect?

    Show answer

    The number of incompatibilities grows faster than linearly, roughly with the square of the number of substitutions.

  6. In birds, which hybrid sex is usually sterile according to Haldane’s rule?

    Show answer

    Females, which are ZW.

  7. Why does polyploidy create instant isolation?

    Show answer

    Crosses with the parent species produce mostly sterile offspring with an odd number of chromosome sets.

  8. What is reinforcement?

    Show answer

    Selection against unfit hybrids strengthens prezygotic isolation in areas of overlap.

  9. A cline is 1 km wide and σ = 100 m. Estimate s.

    Show answer

    √s = 0.1, so s ≈ 0.01.

  10. A clade grew from one to 100 species in 10 million years. Estimate the net diversification rate.

    Show answer

    ln 100/10 ≈ 0.46 per million years.

Lesson quiz · 5 questions

Score 60% or more to complete this lesson.

Take the quiz