Learning without labels: unsupervised and reinforcement learning

Most of the machine learning in Lessons 5 and 6 was supervised: every training example came with the right answer. Labels are expensive, though, and much of what intelligent systems need to learn has no labels at all. Unsupervised learning finds structure in unlabeled data: groups of similar customers, unusual sensor readings, compact representations of images or words. Self-supervised learning, a powerful variant, creates its own labels from the data and is how large language models are pretrained. Reinforcement learning learns from trial, error and reward, the way animals learn to find food or a child learns to ride a bicycle, and it has produced game-playing champions, robot controllers and the reasoning abilities of recent AI models. This lesson explains these three ways of learning.

Clustering

Clustering groups data points so that points in the same group are similar and points in different groups are not. Businesses cluster customers into segments; biologists cluster genes with similar activity; operations teams cluster alarms to find common causes. The best-known algorithm, k-means, works in a few simple steps:

1. Choose the number of clusters \(k\) and pick \(k\) starting centres.

2. Assign each point to its nearest centre.

3. Move each centre to the average of the points assigned to it.

4. Repeat steps 2 and 3 until the assignments stop changing.

The algorithm minimizes the total squared distance from points to their cluster centres (the within-cluster sum of squares). Because it can get stuck in poor solutions, it is run several times from different starting points and the best result kept.

import numpy as np

rng = np.random.default_rng(0)
means = np.array([[2, 2], [8, 3], [5, 8]])
X = np.vstack([c + rng.normal(0, 1.0, (50, 2)) for c in means])

for k in [2, 3, 4]:
    best = None
    for trial in range(5):
        C = X[rng.choice(len(X), k, replace=False)]
        for it in range(50):
            labels = np.argmin(((X[:, None] - C[None]) ** 2).sum(-1), axis=1)
            newC = np.array([X[labels == j].mean(axis=0) for j in range(k)])
            if np.allclose(newC, C):
                break
            C = newC
        inertia = float(((X - C[labels]) ** 2).sum())
        if best is None or inertia < best[0]:
            best = (inertia, C, labels)
    inertia, C, labels = best
    print(f"k = {k}: within-cluster sum of squares {inertia:7.1f}  cluster sizes {np.bincount(labels)}")
    if k == 3:
        print("   means found:", np.round(C[np.argsort(C[:, 0])], 1).tolist())

k = 2: within-cluster sum of squares  1186.3  cluster sizes [100  50]
k = 3: within-cluster sum of squares   302.3  cluster sizes [50 50 50]
   means found: [[2.0, 2.2], [4.6, 8.1], [7.9, 3.0]]
k = 4: within-cluster sum of squares   257.4  cluster sizes [50 21 50 29]

With the right number of clusters (k = 3), k-means recovers the three groups and their centres almost exactly. With k = 2, two groups are forced together and the within-cluster sum of squares is four times larger; with k = 4, one group is split, giving only a small further improvement. Plotting this sum against k and looking for the "elbow" where improvement slows is a common way to choose k, though real data often has no obvious answer.

Dimensionality reduction and anomaly detection

Data often has many features (hundreds of sensor readings, thousands of pixels), many of them correlated. Dimensionality reduction finds a smaller set of features that captures most of the variation. Principal component analysis (PCA) finds the directions along which data varies most; projecting data onto the first two or three components lets people visualize high-dimensional data and lets models train faster. Neural networks called autoencoders learn compressed representations by squeezing data through a narrow middle layer and reconstructing it.

Anomaly detection finds data points that do not fit the usual pattern: fraudulent transactions, failing equipment, cyberattacks or manufacturing defects. A simple approach models what normal data looks like and flags points far from it; an autoencoder trained on normal data reconstructs normal points well and anomalies poorly. Because anomalies are rare, base-rate effects (Lesson 4) make false alarms a constant challenge.

Self-supervised learning

Self-supervised learning creates labels automatically from the structure of the data itself. A language model is trained to predict the next word in a sentence, or a missing word, using text from the internet: the "label" is simply the word that actually came next. An image model may learn by predicting a masked part of an image, or by learning that two altered versions of the same photo belong together. Because no human labelling is needed, self-supervised learning can use enormous datasets, and the representations it learns transfer to many tasks. It is the basis of today's foundation models: a model pretrained on vast unlabeled data and then adapted, with small amounts of labeled data or feedback, to specific uses (Lesson 9).

Reinforcement learning

In reinforcement learning (RL), an agent interacts with an environment over time. At each step it observes a state, chooses an action, and receives a reward and a new state. Its goal is to learn a policy (a rule for choosing actions) that maximizes the total reward it collects over time. No one tells the agent the right action; it must discover which actions lead to reward, often only after a delay.

Future rewards are usually discounted by a factor \(\gamma\) between 0 and 1, so that a reward now counts more than the same reward later:

Q-learning, a classic RL algorithm, learns the value \(Q(s, a)\) of taking action \(a\) in state \(s\) and then acting well afterwards. After each step it nudges its estimate toward the reward received plus the discounted value of the best action in the next state:

import numpy as np

rng = np.random.default_rng(1)
n_states, goal, trap = 7, 6, 0
Q = np.zeros((n_states, 2))
alpha, gamma, eps = 0.5, 0.9, 0.2


def step(s, a):
    s2 = s + (1 if a == 1 else -1)
    if s2 == goal:
        return s2, 10.0, True
    if s2 == trap:
        return s2, -10.0, True
    return s2, -1.0, False


for episode in range(1, 301):
    s, total, done = 3, 0.0, False
    while not done:
        a = rng.integers(2) if rng.random() < eps else int(np.argmax(Q[s]))
        s2, r, done = step(s, a)
        target = r if done else r + gamma * Q[s2].max()
        Q[s, a] += alpha * (target - Q[s, a])
        s, total = s2, total + r
    if episode in (1, 10, 50, 300):
        print(f"episode {episode:>3}: total reward {total:5.1f}")
policy = ["left" if Q[s, 0] > Q[s, 1] else "right" for s in range(1, 6)]
print("learned policy for states 1-5:", policy)
print("Q-values (left, right):", np.round(Q[1:6], 2).tolist())

episode   1: total reward  -4.0
episode  10: total reward   8.0
episode  50: total reward   8.0
episode 300: total reward   8.0
learned policy for states 1-5: ['right', 'right', 'right', 'right', 'right']
Q-values (left, right): [[-5.0, 1.54], [-1.98, 4.58], [3.12, 6.2], [4.58, 8.0], [6.2, 10.0]]

The agent starts knowing nothing, wanders, and occasionally falls into the trap or reaches the goal. Within about ten episodes it has learned to walk straight to the goal, and its Q-values show that moving right is worth more in every state, with values rising toward the goal. The same algorithm, combined with deep neural networks to estimate Q-values from screen images, let DeepMind's DQN learn to play dozens of Atari games at human level in 2015.

Exploration versus exploitation

A reinforcement learning agent faces a dilemma: should it exploit what it already knows works, or explore something new that might be better? Pure exploitation can lock in a mediocre strategy; pure exploration never benefits from learning. A simple approach, used in the code, is epsilon-greedy: act on the best-known action most of the time, but choose randomly with a small probability \(\epsilon\). People face the same trade-off when deciding whether to return to a favourite restaurant or try a new one.

Where reinforcement learning is used

• Games: AlphaGo and AlphaZero (Lesson 3), and agents that reached top human level in complex video games such as StarCraft II and Dota 2.
• Robotics: learning to walk, grasp and manipulate, usually trained in simulation and transferred to real robots.
• Control and operations: DeepMind reported in 2016 that a machine learning system reduced the energy used for cooling Google data centres by up to 40%, and later systems controlled cooling directly; RL is also studied for power grids, traffic signals and industrial processes.
• Recommendation and advertising: learning which content to show from user responses.
• Language models: reinforcement learning from human feedback (RLHF) teaches assistants to give helpful, safe answers, and reinforcement learning on problems with checkable answers produced reasoning models in 2024 and 2025 (Lesson 9).

Reinforcement learning has pitfalls. It can need huge numbers of trials, which is why simulation is so important. And agents optimize exactly the reward they are given, which may differ from what designers intended: an agent in a boat-racing game famously learned to spin in circles collecting bonus points instead of finishing the race. This problem, called reward hacking or specification gaming, is central to AI safety (Lesson 11).