Data for language models

The scaling laws of Lesson 3 treat training data as a number of tokens, but every token comes from somewhere, and what those tokens say determines what a model can learn. Two models of the same size trained with the same compute can differ greatly in ability because of their data. Leading developers now treat data as their most important and closely guarded asset, and much of the progress in open models since 2023 has come from better data rather than new architectures. This lesson follows pretraining data from its sources through extraction, filtering and deduplication, looks at how sources are mixed, examines what happens as the supply of human-written text runs short, and covers benchmark contamination and the documentation that responsible data practice requires.

Where the data comes from

Pretraining corpora for current frontier models contain tens of trillions of tokens; Meta reported training Llama 3 on more than 15 trillion tokens in 2024. The main sources are:

• Web pages. Common Crawl, a non-profit that has crawled the web since 2008 and publishes regular snapshots of billions of pages, is the starting point for most open datasets; developers also run their own crawlers.
• Code. Public software repositories provide code in hundreds of languages; datasets such as The Stack include only permissively licensed code and honour opt-out requests.
• Books, academic papers and encyclopedias, which contribute long, carefully edited text.
• Forums and question-and-answer sites, which contribute dialogue and explanations.
• Licensed data. Since 2023, developers have signed licensing agreements with news publishers, stock photo libraries and online communities for access to their content.
• Synthetic data generated by other models, discussed later in this lesson.

Most web text is low in quality for training purposes: navigation menus, advertising, duplicated templates, spam and machine-generated pages. Of the raw HTML in a crawl, only a small fraction survives a modern pipeline as training text.

Extraction and filtering

Lesson 1 listed the stages of a data pipeline. Each involves real engineering choices:

• Text extraction. Tools that strip HTML markup must keep the main content, headings, lists, tables, code blocks and mathematics, while discarding menus and boilerplate. Poor extraction loses structure that models need to learn formatting.
• Language identification. A fast classifier assigns each document a language and a confidence score. Multilingual models need careful thresholds, because classifiers are less accurate for low-resource languages and for documents that mix languages.
• URL and domain filtering. Blocklists remove adult content, known spam and malware domains.
• Heuristic quality filters. Simple rules remove documents that are too short or too long, have too few alphabetic characters, too many repeated lines or n-grams, or too few common words. Rules of this kind, published with the Gopher model in 2021, remain widely used.
• Model-based quality filters. A classifier trained to recognize high-quality text scores every document, and only the best are kept. For the FineWeb-Edu dataset released in 2024, for example, a large language model rated a sample of web pages for educational value, and a small classifier trained on those ratings filtered the full corpus; models trained on the filtered data learned markedly faster on knowledge and reasoning benchmarks.
• Safety and privacy filters. Classifiers and pattern matching remove toxic content and personal information such as email addresses, phone numbers and identification numbers.

Every filter involves trade-offs. Aggressive quality filtering can remove dialects, informal writing and minority viewpoints along with spam; toxicity filters trained on biased labels can disproportionately remove text written by or about some groups. Pipelines are therefore evaluated not only by the benchmark scores of models trained on them, but also by inspecting what they remove.

Deduplication

The web is full of duplicates: the same article syndicated across sites, templates repeated on millions of pages, and boilerplate such as licence text and cookie notices. Training on duplicates wastes compute, increases memorization (and with it the risk of reproducing copyrighted or private text verbatim), and can leak evaluation data into training. Research published in 2021 showed that deduplicating training data reduced memorized output roughly tenfold while improving training efficiency.

Exact duplicates are easy to find by hashing documents or lines. Near-duplicates, which differ by small edits, dates or formatting, need a similarity measure. The standard approach represents each document as a set of overlapping word n-grams (shingles) and measures the Jaccard similarity of two sets, the size of their intersection divided by the size of their union. Comparing every pair of documents in a corpus of billions is impossible, so MinHash and locality-sensitive hashing (LSH) are used:

• MinHash computes many hash functions over each document's shingles and keeps the minimum value of each. The probability that two documents share the same minimum for one hash function equals their Jaccard similarity, so the fraction of matching minimums estimates it.
• LSH groups the MinHash values into bands. Two documents become candidate duplicates if all values in any one band match. With \(b\) bands of \(r\) values, documents with similarity \(s\) are flagged with probability \(1 - (1 - s^r)^b\), an S-shaped curve that separates similar and dissimilar pairs sharply.

import hashlib
import random

# Near-duplicate detection with MinHash over word 5-gram shingles
def shingles(text, n=5):
    w = text.lower().split()
    return {" ".join(w[i:i + n]) for i in range(len(w) - n + 1)}

def h(s, seed):
    return int(hashlib.md5(f"{seed}:{s}".encode()).hexdigest()[:12], 16)

def minhash(sh, k=128):
    return [min(h(s, seed) for s in sh) for seed in range(k)]

def jaccard(a, b):
    return len(a & b) / len(a | b)

random.seed(1)
vocab = [f"w{i}" for i in range(2000)]
base = [random.choice(vocab) for _ in range(300)]
near = base.copy()
for i in random.sample(range(300), 12):        # change 4% of words
    near[i] = random.choice(vocab)
other = [random.choice(vocab) for _ in range(300)]
docs = {"original": " ".join(base), "edited copy": " ".join(near), "unrelated": " ".join(other)}

S = {k: shingles(v) for k, v in docs.items()}
M = {k: minhash(v) for k, v in S.items()}
for a, b in [("original", "edited copy"), ("original", "unrelated")]:
    true_j = jaccard(S[a], S[b])
    est = sum(x == y for x, y in zip(M[a], M[b])) / len(M[a])
    print(f"{a} vs {b}: Jaccard {true_j:.3f}, MinHash estimate {est:.3f}")

# Banding (LSH): 128 hashes in 16 bands of 8; documents become candidates if any band matches exactly
b, r = 16, 8
for s in (0.3, 0.5, 0.7, 0.8, 0.9):
    p = 1 - (1 - s ** r) ** b
    print(f"similarity {s:.1f}: probability flagged as candidate {p:.3f}")

original vs edited copy: Jaccard 0.696, MinHash estimate 0.742
original vs unrelated: Jaccard 0.000, MinHash estimate 0.000
similarity 0.3: probability flagged as candidate 0.001
similarity 0.5: probability flagged as candidate 0.061
similarity 0.7: probability flagged as candidate 0.613
similarity 0.8: probability flagged as candidate 0.947
similarity 0.9: probability flagged as candidate 1.000

The output shows two useful facts. Changing only 4 percent of the words in a document reduces its 5-gram Jaccard similarity to about 0.7, because each changed word breaks up to five shingles, so near-duplicate thresholds must be set with the shingle length in mind. And with 16 bands of 8 hashes, pairs with similarity 0.8 or more are almost always caught, while pairs below 0.5 are rarely examined, which keeps the number of comparisons manageable.

Mixing sources

After cleaning, the sources are combined in chosen proportions, the data mixture. Mixtures are tuned because different sources teach different skills: code improves reasoning and structured output even for non-programming tasks, mathematics improves quantitative ability, multilingual text extends coverage, and books and papers add knowledge and long-range coherence. High-quality sources are often upsampled (repeated more than their share of raw tokens), while abundant web text is downsampled.

Mixtures also change during training. A common pattern is a long main phase on a broad mixture, followed by a shorter annealing phase near the end, when the learning rate is decreased, using a higher proportion of carefully selected text, mathematics, code and instruction-like data. Mixtures are chosen by training small proxy models on candidate mixtures and comparing their performance, and some methods optimize the weights automatically.

Running short of text

The amount of high-quality public human-written text is finite. A widely cited 2024 analysis by the research group Epoch AI estimated the effective stock of public text at around 300 trillion tokens, and projected that if trends continued, the largest training runs would use it all at some point between 2026 and 2032. Developers have several responses.

Repeating data is the simplest. A 2023 study of data-constrained scaling found that training on the same data for up to about four epochs is almost as good as training on fresh data, but the value of each further repetition declines, and beyond a few dozen epochs extra passes add almost nothing. The study fitted a formula for the effective amount of data that repeated tokens are worth:

where \(U\) is the number of unique tokens, \(R\) the number of repetitions beyond the first epoch and \(R^{*} \approx 15\) a fitted constant.

import math

# 1. Contamination check: share of each benchmark item covered by 13-gram matches in the corpus
def ngrams(text, n=13):
    w = text.lower().split()
    return {tuple(w[i:i + n]) for i in range(len(w) - n + 1)}

corpus = ("forum post: here is the answer to the puzzle a farmer has 17 sheep and all but 9 run away "
          "how many sheep does the farmer have left the answer is 9 because all but 9 ran away "
          "unrelated text about cooking pasta in salted water for ten minutes until tender")
benchmark = {
    "item 1": "a farmer has 17 sheep and all but 9 run away how many sheep does the farmer have left",
    "item 2": "a train leaves the station at 3 pm moving at 80 km per hour how far has it gone by 5 pm",
}
C = ngrams(corpus)
for name, q in benchmark.items():
    g = ngrams(q)
    hit = len(g & C) / len(g)
    flag = "CONTAMINATED" if hit > 0.5 else "clean"
    print(f"{name}: {hit:.0%} of 13-grams found in corpus -> {flag}")

# 2. Value of repeated data (Muennighoff et al. 2023): effective tokens from U unique tokens seen R extra times
R_star = 15.4
def effective(U, epochs):
    R = epochs - 1
    return U + U * R_star * (1 - math.exp(-R / R_star))

U = 2e12                      # 2 trillion unique tokens
print()
for ep in (1, 2, 4, 8, 16, 40):
    eff = effective(U, ep)
    print(f"{ep:2d} epochs: {ep * U / 1e12:5.0f}T tokens seen, worth {eff / 1e12:5.1f}T unique "
          f"({eff / (ep * U):.0%} value per token)")

item 1: 100% of 13-grams found in corpus -> CONTAMINATED
item 2: 0% of 13-grams found in corpus -> clean

 1 epochs:     2T tokens seen, worth   2.0T unique (100% value per token)
 2 epochs:     4T tokens seen, worth   3.9T unique (98% value per token)
 4 epochs:     8T tokens seen, worth   7.5T unique (93% value per token)
 8 epochs:    16T tokens seen, worth  13.3T unique (83% value per token)
16 epochs:    32T tokens seen, worth  21.2T unique (66% value per token)
40 epochs:    80T tokens seen, worth  30.4T unique (38% value per token)

The second part of the output shows the diminishing returns: four epochs of 2 trillion unique tokens are worth about 7.5 trillion fresh tokens (93 percent of their face value), but 40 epochs are worth only about 30 trillion, 38 percent.

Synthetic data is the other main response. Models generate textbook-style explanations, exercises, rephrasings of web pages in clearer language, translations, and above all mathematics and code whose correctness can be checked automatically by running tests or verifying answers. Microsoft's Phi series of small models, trained heavily on synthetic textbook-quality data, showed in 2023 and 2024 that such data can produce strong small models. Synthetic data also underlies the reasoning training of Lesson 5.

Synthetic data carries risks. Generated text inherits the errors and biases of the model that produced it and tends to be less diverse than human writing. A 2024 study in Nature showed that models trained recursively on the output of previous generations, without fresh human data, progressively lose the rare parts of the distribution and degrade, a process called model collapse. Mixing synthetic data with real data, verifying it where possible, and deliberately generating diverse content reduce the risk. As AI-generated text spreads across the web, filtering it out of future crawls, or at least knowing how much is present, has become part of data curation.

Benchmark contamination

If test questions from a benchmark appear in the training data, a model's score measures memory rather than ability. Contamination happens easily: benchmarks are published on the web, discussed in forums, and copied into study materials, all of which end up in crawls. Contaminated scores can mislead researchers about progress and customers about capability.

Developers detect contamination by searching the training data for overlapping sequences of words, typically 8 to 13 words long, from each test item, and either remove the matching training documents or report results on the clean subset of the benchmark. Benchmark creators can embed a unique canary string in their files so that developers can find and exclude them. Other defences include private held-out test sets, benchmarks that are regularly refreshed with new problems written after models' training cut-off dates (for example, coding problems from recent programming contests), and comparing performance on a benchmark with performance on a newly written equivalent. Lesson 6 discusses evaluation more broadly.

Documentation and responsibility

Because data shapes what models know and how they behave, responsible practice includes documenting it. Datasheets for datasets and data cards record where data came from, how it was collected and filtered, what it contains, known gaps and biases, licensing, and intended uses. Model cards summarize the training data along with evaluation results. Regulators increasingly expect such documentation: the European Union's AI Act requires providers of general-purpose AI models to publish a sufficiently detailed summary of their training content.

Data practices also raise legal and ethical questions: whether training on copyrighted work is lawful (discussed in Lesson 9), whether people whose writing or personal information appears in a crawl have consented, and whether website owners' wishes are respected. Most major crawlers now honour robots.txt instructions that exclude them, and many publishers have used those instructions to block AI crawlers since 2023, which itself shrinks the available pool of high-quality text. Data that includes personal information must be handled in line with privacy law, and developers increasingly offer ways for people to request removal of their information.