Labelling and dataset documentation

Supervised learning needs labels: the right answer for each training example. Sometimes labels come for free, recorded by the process itself (whether a customer churned, whether a loan defaulted). More often they must be created by people: annotators who mark which emails are spam, draw boxes around pedestrians, transcribe speech or rate which of two chatbot answers is better. Labels define what the model learns, so their quality sets a ceiling on the model's quality, and their cost often dominates a project's budget.

This lesson covers how to design a labelling task, measure agreement between annotators, combine and clean labels, reduce labelling cost, split labelled data correctly, protect privacy, and document a data set so that others can understand and use it responsibly.

Designing a labelling task

A labelling project starts with a written guideline that defines each label precisely, with examples of clear cases, borderline cases and how to decide them. Vague guidelines produce inconsistent labels: "is this review negative?" leaves annotators to decide what to do with mixed reviews, sarcasm and reviews about shipping rather than the product. A good process:

• Defines the label set to be exhaustive and mutually exclusive, often with an "unsure" or "other" option so that annotators are not forced to guess.
• Runs a pilot: several annotators label the same small batch, disagreements are discussed, and the guideline is revised until agreement is acceptable.
• Includes gold questions: items with known answers mixed into the work to check each annotator's accuracy over time.
• Measures agreement continuously and routes disagreements to an expert for adjudication.
• Treats annotators as skilled workers: clear instructions, fair pay, reasonable throughput targets and, for disturbing content, protections such as limits on exposure and access to support.

Labels can come from in-house experts (accurate but expensive, essential in medicine and law), managed annotation vendors, crowdsourcing platforms (cheap and fast but noisier), or from models, including large language models used as first-pass labellers whose output is checked by people.

Measuring agreement

When two annotators label the same items, the simplest measure is the proportion of items on which they agree. But some agreement happens by chance: if both label 90% of items "not spam", they will agree on many items even when guessing. Cohen's kappa corrects for this:

where \(p_o\) is the observed agreement and \(p_e\) the agreement expected by chance from each annotator's label frequencies. Kappa is 1 for perfect agreement and 0 for agreement no better than chance. Rough conventions call values above about 0.8 strong and below about 0.4 weak, but what is acceptable depends on the task; subjective tasks such as rating humour never reach the agreement of objective ones. For more than two annotators, Fleiss' kappa and Krippendorff's alpha generalize the idea.

import numpy as np
from sklearn.metrics import cohen_kappa_score, confusion_matrix

a = np.array(["spam"] * 20 + ["not"] * 65 + ["spam"] * 10 + ["not"] * 5)
b = np.array(["spam"] * 20 + ["not"] * 65 + ["not"] * 10 + ["spam"] * 5)

print(confusion_matrix(a, b, labels=["spam", "not"]))
po = (a == b).mean()
pe = sum((a == k).mean() * (b == k).mean() for k in ["spam", "not"])
print(f"observed {po:.3f}, chance {pe:.3f}, kappa {(po - pe) / (1 - pe):.3f}")
print(f"sklearn kappa {cohen_kappa_score(a, b):.3f}")

[[20 10]
 [ 5 65]]
observed 0.850, chance 0.600, kappa 0.625
sklearn kappa 0.625

Combining labels

When several annotators label each item, their labels must be combined. Majority vote is the simplest rule. It can be improved by weighting each annotator by their accuracy on gold questions, or by statistical models (such as the Dawid-Skene model) that estimate each annotator's error pattern and each item's true label together. Items on which annotators split evenly are often genuinely ambiguous; rather than forcing a single label, they can be adjudicated by an expert, removed, or kept with soft labels (for example 0.6 spam, 0.4 not spam), which tell the model the item is uncertain.

import pandas as pd

votes = pd.DataFrame({
    "item":      [1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4],
    "annotator": ["a", "b", "c"] * 4,
    "label":     ["pos", "pos", "neg", "neg", "neg", "neg", "pos", "neg", "neu", "neu", "pos", "pos"],
})
wide = votes.pivot(index="item", columns="annotator", values="label")
counts = votes.groupby("item")["label"].value_counts().unstack(fill_value=0)
wide["majority"] = counts.idxmax(axis=1).where(counts.max(axis=1) >= 2, "ADJUDICATE")
wide["agreement"] = (counts.max(axis=1) / counts.sum(axis=1)).round(2)
print(wide)

annotator    a    b    c    majority  agreement
item
1          pos  pos  neg         pos       0.67
2          neg  neg  neg         neg       1.00
3          pos  neg  neu  ADJUDICATE       0.33
4          neu  pos  pos         pos       0.67

Label noise

Even with good processes, some labels are wrong. Studies of widely used benchmark data sets have found error rates of a few percent in their test sets, enough to change which model appears best. The effect of noise depends on its type: random noise, where any label is equally likely to be flipped, mostly slows learning and lowers accuracy gradually; systematic noise, where certain classes or groups are consistently mislabelled, teaches the model the annotators' mistakes.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=6000, n_features=10, n_informative=5, random_state=0)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.5, random_state=0)
rng = np.random.default_rng(0)

for rate in [0.0, 0.1, 0.2, 0.3, 0.4]:
    noisy = ytr.copy()
    flip = rng.random(len(noisy)) < rate
    noisy[flip] = 1 - noisy[flip]
    acc = LogisticRegression(max_iter=1000).fit(Xtr, noisy).score(Xte, yte)
    print(f"{rate:.0%} of training labels flipped -> clean test accuracy {acc:.3f}")

0% of training labels flipped -> clean test accuracy 0.837
10% of training labels flipped -> clean test accuracy 0.836
20% of training labels flipped -> clean test accuracy 0.829
30% of training labels flipped -> clean test accuracy 0.825
40% of training labels flipped -> clean test accuracy 0.805

With enough data, a simple model tolerates a surprising amount of random noise, because the flips average out. Flexible models that can memorize their training data, such as large neural networks, are more affected, and noise in the test set is worse still, because it corrupts the measurement itself.

Finding label errors

Mislabelled items can be found by training a model with cross-validation (so that each item is predicted by a model that did not see it) and flagging items where the model is confident that the given label is wrong. This is the idea behind confident learning and tools such as cleanlab. Flagged items are then reviewed by people; the model only suggests where to look.

import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_predict

X, y = make_classification(n_samples=2000, n_features=8, n_informative=5,
                           class_sep=1.5, random_state=1)
rng = np.random.default_rng(1)
wrong = rng.choice(len(y), size=60, replace=False)      # corrupt 60 labels (3%)
y_noisy = y.copy()
y_noisy[wrong] = 1 - y_noisy[wrong]

proba = cross_val_predict(LogisticRegression(max_iter=1000), X, y_noisy, cv=5, method="predict_proba")
p_given = proba[np.arange(len(y)), y_noisy]              # model probability of the given label
suspects = np.argsort(p_given)[:80]                      # 80 least plausible labels

found = np.isin(suspects, wrong).sum()
print(f"reviewed 80 items ({80 / len(y):.0%} of data), found {found} of 60 corrupted labels")

reviewed 80 items (4% of data), found 35 of 60 corrupted labels

Reviewing the 80 most suspicious items, 4% of the data, found more than half of the corrupted labels; a random review of 80 items would be expected to find only about 2 or 3 (3% of 80). Ranking by model disagreement concentrates human review where errors are likely.

Reducing labelling cost

Several techniques reduce how many labels are needed:

• Active learning: train a model on a small labelled set, then ask annotators to label the unlabelled items the model is least certain about, which are often more informative than random items.
• Weak supervision: write labelling functions (keyword rules, heuristics, existing databases) that each label some items noisily, and combine their votes statistically. Programmatic labels are cheap and can be revised by editing the rules.
• Pre-labelling: a model, often a large language model, proposes labels that annotators accept or correct, which is faster than labelling from scratch. Annotators tend to accept proposals, so pre-labelling must be audited against independent labels to make sure it is not biasing the result.
• Transfer and self-supervised learning: models pretrained on large unlabelled data sets need far fewer labelled examples to reach a given accuracy.

Splitting labelled data

Labelled data is divided into training, validation and test sets. The test set must be held back until the end and used only to report final performance. Two rules matter most:

• Split by group when records are related. All records from the same patient, user, device, document or session go into the same split; otherwise the model is rewarded for recognizing individuals it has already seen. Scikit-learn provides GroupKFold and GroupShuffleSplit for this.
• Split by time when the model will predict the future. Train on earlier data and test on later data, so that the test mimics deployment. A random split of time-ordered data lets the model "see the future".

import numpy as np
import pandas as pd
from sklearn.model_selection import GroupShuffleSplit, train_test_split

rng = np.random.default_rng(2)
patients = np.repeat(np.arange(200), rng.integers(1, 8, 200))   # 1-7 scans per patient
df = pd.DataFrame({"patient": patients})

tr, te = train_test_split(df, test_size=0.2, random_state=0)
shared = len(set(tr.patient) & set(te.patient))
print(f"random split: {len(te)} test scans, {shared} patients in both sets")

gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=0)
tr_idx, te_idx = next(gss.split(df, groups=df["patient"]))
shared = len(set(df.patient.iloc[tr_idx]) & set(df.patient.iloc[te_idx]))
print(f"group split:  {len(te_idx)} test scans, {shared} patients in both sets")

random split: 161 test scans, 98 patients in both sets
group split:  155 test scans, 0 patients in both sets

Privacy in labelled data

Labelled data sets often contain personal information: names in support tickets, faces in images, health details in clinical notes. Before data is shared with annotators or published, it should be de-identified as far as the purpose allows. Direct identifiers (names, addresses, account numbers) are removed or replaced; quasi-identifiers, such as postal code, birth date and sex, which can identify people in combination, are generalized (a birth year instead of a date, the first three characters of a postal code).

One way to check the risk of re-identification is k-anonymity: a table is k-anonymous if every combination of quasi-identifier values is shared by at least k records. It does not guarantee privacy, since a group of k people who all share a sensitive value still reveals it, and stronger methods such as differential privacy add calibrated noise to give mathematical guarantees.

import pandas as pd

df = pd.DataFrame({
    "birth_year": [1980, 1980, 1981, 1980, 1981, 1981, 1982, 1980],
    "postal3":    ["T2N", "T2N", "T2N", "T2N", "T2N", "T2N", "T3A", "T2N"],
    "sex":        ["F", "F", "M", "F", "M", "M", "F", "F"],
    "diagnosis":  ["flu", "asthma", "flu", "flu", "diabetes", "flu", "asthma", "flu"],
})
qi = ["birth_year", "postal3", "sex"]
sizes = df.groupby(qi).size()
print(sizes)
print("k =", sizes.min())

df["birth_decade"] = (df["birth_year"] // 10 * 10).astype(str) + "s"
k2 = df.groupby(["birth_decade", "postal3", "sex"]).size().min()
df["region"] = "T"
k3 = df.groupby(["birth_decade", "region", "sex"]).size().min()
print("k after generalizing year:", k2, "| and postal code:", k3)

birth_year  postal3  sex
1980        T2N      F      4
1981        T2N      M      3
1982        T3A      F      1
dtype: int64
k = 1
k after generalizing year: 1 | and postal code: 3

The one woman born in 1982 with postal prefix T3A is unique (k = 1), and generalizing birth years to decades does not help because her postal prefix still sets her apart. Generalizing the postal code as well puts every record in a group of at least three. Each generalization loses detail, so the right level is a trade-off between privacy risk and usefulness for the task.

In Canada, privacy laws apply to information about an identifiable individual, so properly de-identified data can fall outside some obligations, but the bar is high. Quebec's Law 25 distinguishes de-identified information (which can no longer directly identify a person but still could with other information, and remains personal information) from anonymized information (from which identification is irreversibly impossible, assessed under generally accepted best practices). Provincial health information laws, such as Ontario's PHIPA, set additional rules for health data.

Documenting a data set

A data set without documentation is a liability: future users cannot tell what it represents, how it was labelled or what it should not be used for. Two widely used formats are:

• Datasheets for datasets: a set of questions, modelled on electronics datasheets, covering motivation (why the data set was created and by whom), composition (what the records represent, how many, what is missing, whether there is personal information), collection process, preprocessing and labelling, intended and inappropriate uses, distribution and licence, and maintenance.
• Data cards and model cards: short structured summaries, often published with a data set or model, giving its intended use, sources, label definitions, known limitations, performance broken down by relevant groups, and ethical considerations. A model card documents the model trained on the data, including where it was evaluated and where it should not be used.

Documentation should also record the licence (for example CC BY 4.0, which permits any use with attribution, or CC BY-NC, which prohibits commercial use), the version, and how the data set was split, so that results reported by different teams are comparable. Lesson 8 covers versioning data alongside code.

Lesson 7 goes further into privacy: governance, the main privacy laws, linkage attacks, pseudonymization and differential privacy.