Text classification and sequence labelling

Two task families account for a large share of practical NLP. Text classification assigns a label to a whole document or sentence: spam or not, positive or negative, which department should handle a support ticket, whether a post violates a policy. Sequence labelling assigns a label to each token: part-of-speech tags, named entities (people, organizations, places), or slots in a voice command. Large language models can now perform both by prompting, but trained classifiers and taggers remain widely used because they are fast, cheap, auditable and accurate when labelled data exist. This lesson covers the classical and neural methods for both, the labelling schemes and decoding constraints for sequences, how to build data sets, and how to choose between training a model and prompting a large one.

Text classification as supervised learning

A classifier maps a document \(d\) to one of \(K\) classes. With the bag-of-words or TF-IDF vectors of Lesson 1, any standard classifier can be used. Two classical choices are instructive:

• Multinomial naive Bayes is a generative model: it estimates the probability of each word given each class from counts (with add-one smoothing) and classifies by Bayes' theorem, assuming words are independent given the class: \(\hat c = \arg\max_c \log P(c) + \sum_i \log P(w_i \mid c)\). It trains in one pass and works well with little data.
• Logistic regression is a discriminative model: it learns a weight for each word directly to separate the classes, \(P(y = 1 \mid \mathbf{x}) = \sigma(\mathbf{w}^\top\mathbf{x} + b)\), usually with L2 regularization. It handles correlated features better and is usually more accurate with enough data. Its weights are directly interpretable as evidence for or against each class.

import numpy as np
from collections import Counter

rng = np.random.default_rng(1)
pos_words = "great excellent love wonderful fast helpful friendly perfect easy recommend".split()
neg_words = "terrible awful hate slow broken rude useless refund poor disappointed".split()
neutral = "the service product order delivery staff app was is it and very really my of".split()
def review(label):
    words = list(rng.choice(neutral, 6))
    words += list(rng.choice(pos_words if label else neg_words, 2))
    if rng.random() < 0.3:                                   # some noise: an opposite-sentiment word
        words += list(rng.choice(neg_words if label else pos_words, 1))
    if rng.random() < 0.15:                                  # negation flips meaning
        words = ["not"] + words; label = 1 - label
    rng.shuffle(words)
    return words, label
train = [review(int(rng.random() < 0.5)) for _ in range(400)]
test = [review(int(rng.random() < 0.5)) for _ in range(400)]
vocab = sorted({w for d, _ in train for w in d}); idx = {w: i for i, w in enumerate(vocab)}
def vec(d):
    x = np.zeros(len(vocab))
    for w in d:
        if w in idx: x[idx[w]] += 1
    return x
Xtr = np.array([vec(d) for d, _ in train]); ytr = np.array([y for _, y in train])
Xte = np.array([vec(d) for d, _ in test]); yte = np.array([y for _, y in test])

# Multinomial naive Bayes with add-one smoothing
logprior = np.log([np.mean(ytr == c) for c in (0, 1)])
counts = np.array([Xtr[ytr == c].sum(0) + 1 for c in (0, 1)])
loglik = np.log(counts / counts.sum(1, keepdims=True))
nb_pred = np.argmax(Xte @ loglik.T + logprior, 1)

# Logistic regression by gradient descent with L2 regularization
w = np.zeros(len(vocab)); b = 0.0
for _ in range(500):
    p = 1 / (1 + np.exp(-(Xtr @ w + b)))
    w -= 0.1 * (Xtr.T @ (p - ytr) / len(ytr) + 0.01 * w); b -= 0.1 * np.mean(p - ytr)
lr_pred = (Xte @ w + b > 0).astype(int)
print(f"naive Bayes accuracy {np.mean(nb_pred == yte):.3f}; logistic regression accuracy {np.mean(lr_pred == yte):.3f}")
top = np.argsort(w)
print("most negative weights:", [str(vocab[i]) for i in top[:5]])
print("most positive weights:", [str(vocab[i]) for i in top[-5:]])
neg = idx["not"]
print(f"weight learned for the word not: {w[neg]:+.2f} (a bag of words cannot express that it flips the words around it)")

naive Bayes accuracy 0.855; logistic regression accuracy 0.835
most negative weights: ['hate', 'broken', 'rude', 'useless', 'refund']
most positive weights: ['recommend', 'easy', 'love', 'perfect', 'helpful']
weight learned for the word not: +0.15 (a bag of words cannot express that it flips the words around it)

Both classifiers reach about 85% accuracy and learn sensible word weights. Both fail on negation: in a bag of words, "not" is just another word, and its weight cannot express that it reverses the sentiment of the words around it. Adding bigram features ("not great") helps a little; models that represent word order and context, such as recurrent networks (Lesson 4) and transformers (Lesson 6), handle it properly.

For more than two classes, multinomial logistic regression (softmax) generalizes directly. For multi-label problems, where a document can belong to several classes at once (a news article about both sport and business), a separate binary classifier per label is trained, each with its own threshold.

Neural text classifiers

Neural classifiers replace hand-built features with learned representations:

• Averaged embeddings: average the word embeddings of a document (Lesson 2) and feed the result to a classifier. Simple models of this kind, such as fastText, are surprisingly strong and extremely fast.
• Convolutional networks apply filters over windows of consecutive word embeddings, detecting informative phrases regardless of position, followed by max pooling.
• Recurrent networks read the text in order and use the final or pooled hidden state (Lesson 4).
• Fine-tuned transformers: a pretrained encoder such as BERT (Lesson 7) reads the text, and a small classification layer on the special classification token's output is trained on the labelled data, updating all weights. This is the standard approach for high accuracy, typically needing only hundreds to a few thousand labelled examples because the pretrained model already understands language.
• Sentence embeddings plus a linear classifier: encode each text with a pretrained embedding model and train logistic regression on the vectors, a cheap and robust baseline.

Sequence labelling

In sequence labelling, every token receives a label. Common tasks:

• Part-of-speech tagging: noun, verb, adjective and so on, using tag sets such as the Universal Dependencies tags.
• Named entity recognition (NER): identifying spans that name people, organizations, locations, dates, amounts, products, or domain-specific entities such as drugs and genes.
• Slot filling: extracting the arguments of a command, such as the destination and time in "book a flight to Calgary on Friday".
• Chunking and segmentation, and token-level detection of personal information for redaction.

Spans are turned into per-token labels with the BIO scheme: B-X marks the first token of an entity of type X, I-X a token inside it, and O a token outside any entity. "Hydro Quebec opened a site in Saguenay" becomes B-ORG I-ORG O O O O B-LOC. Variants such as BIOES add explicit end and single-token labels.

Models and decoding for sequences

Per-token classifiers ignore dependencies between labels, and can produce invalid sequences, such as I-LOC immediately after O, or I-ORG continuing a B-LOC. Classical sequence models address this:

• Hidden Markov models (AI-320, Lesson 3) model tag transitions and word emissions generatively, decoded with the Viterbi algorithm.
• Conditional random fields (AI-320, Lesson 4) model the tag sequence given the whole input discriminatively, with features of the current word, its neighbours, capitalization, suffixes and adjacent tag pairs.
• Neural taggers compute contextual scores for each token with a bidirectional LSTM (Lesson 4) or a transformer encoder (Lesson 7), often with a CRF layer on top that learns transition scores and enforces valid sequences during Viterbi decoding.

import numpy as np

# Viterbi decoding of BIO tags with transition constraints, as in a CRF output layer
tags = ["O", "B-ORG", "I-ORG", "B-LOC", "I-LOC"]
words = ["Hydro", "Quebec", "opened", "a", "site", "in", "Saguenay", "Lac", "Saint", "Jean"]
# Per-token scores from a tagger (log-potentials); the tagger alone makes inconsistent choices
emit = np.array([
    [0.1, 2.0, 0.5, 0.3, 0.1], [0.2, 0.4, 1.8, 0.9, 0.3], [3.0, 0.1, 0.1, 0.1, 0.1], [3.0, 0.1, 0.1, 0.1, 0.1],
    [2.5, 0.2, 0.1, 0.3, 0.1], [3.0, 0.1, 0.1, 0.1, 0.1], [0.3, 0.2, 0.1, 1.9, 0.4], [0.4, 0.2, 1.2, 0.3, 1.1],
    [0.3, 0.1, 0.9, 0.2, 1.4], [0.5, 0.1, 0.7, 0.2, 1.3]])
allowed = np.zeros((5, 5))
for i, a in enumerate(tags):
    for j, b in enumerate(tags):
        if b.startswith("I-") and not (a != "O" and a[2:] == b[2:]):
            allowed[i, j] = -1e9                          # I-X may only follow B-X or I-X
greedy = [tags[k] for k in emit.argmax(1)]
score = emit[0].copy(); back = []
for t in range(1, len(words)):
    cand = score[:, None] + allowed + emit[t][None, :]
    back.append(cand.argmax(0)); score = cand.max(0)
path = [int(score.argmax())]
for bp in reversed(back): path.append(int(bp[path[-1]]))
best = [tags[k] for k in reversed(path)]
for wd, g, v in zip(words, greedy, best):
    print(f"{wd:9s} greedy {g:6s} constrained {v}")

Hydro     greedy B-ORG  constrained B-ORG
Quebec    greedy I-ORG  constrained I-ORG
opened    greedy O      constrained O
a         greedy O      constrained O
site      greedy O      constrained O
in        greedy O      constrained O
Saguenay  greedy B-LOC  constrained B-LOC
Lac       greedy I-ORG  constrained I-LOC
Saint     greedy I-LOC  constrained I-LOC
Jean      greedy I-LOC  constrained I-LOC

Choosing each token's best tag independently labels "Lac" as I-ORG directly after B-LOC, an invalid sequence that would split the place name. Viterbi decoding with the constraint that I-X may only follow B-X or I-X finds the best valid sequence, tagging "Saguenay Lac Saint Jean" as one location. In a trained CRF layer, the transition scores are learned rather than hand-set, capturing tendencies as well as hard rules.

With subword tokenizers (Lesson 1), a word may be split into several tokens. The usual practice is to predict the label on the first subword of each word and ignore or copy it for the rest, then map predictions back to words for evaluation.

Building labelled data

The quality of a supervised model is bounded by its labels. Building a data set involves:

• Annotation guidelines: precise definitions with examples and edge cases (is "Montreal Canadiens" an organization or a location? does "next Friday" count as a date?). Guidelines evolve as annotators encounter unclear cases.
• Measuring agreement: having several annotators label the same items and computing inter-annotator agreement, such as Cohen's kappa, which corrects raw agreement for chance: \(\kappa = (p_o - p_e)/(1 - p_e)\). Low agreement signals unclear guidelines or a genuinely subjective task.
• Sampling: labelling data representative of the deployment setting, with enough examples of rare classes, possibly using active learning to choose the most informative examples for annotation.
• Weak supervision and LLM labelling: generating noisy labels from heuristic rules or by prompting a large language model, then training a smaller model on them; useful for bootstrapping, with human-labelled test sets to measure real accuracy.
• Splits: train, validation and test sets, split by source, author or time where leakage is possible, so that the test measures generalization.

Train a classifier or prompt a large model?

Large language models (Lesson 8) can classify and extract entities from a prompt with a few examples (in-context learning), and are often good enough without any training data. The choice depends on:

• Volume and cost: classifying millions of items a day is far cheaper and faster with a small fine-tuned model than with calls to a large model.
• Accuracy: with enough labelled data, a fine-tuned model of modest size often matches or beats prompted large models on a fixed, well-defined task; for open-ended, rapidly changing or low-data tasks, prompting wins.
• Latency and deployment: small models run on a CPU or at the edge; large models need GPUs or an external service.
• Control and auditability: a trained classifier's behaviour is fixed and can be thoroughly tested; prompted models can change with model updates.
• Data governance: sensitive data may not be sendable to an external model service (Lesson 12).

A common pattern is to prototype with a prompted large model, use it to label data (with human review), and then train a small model for production, keeping the large model for hard or ambiguous cases.