Learning agents: reinforcement learning

Value iteration needs a model of the environment: the probabilities of every outcome of every action. Most interesting environments do not come with such a model. Reinforcement learning (RL) lets an agent learn good behaviour from experience alone, by acting, observing rewards and adjusting. RL taught computers to play Atari games and Go at superhuman levels, controls robots, cooling systems and recommendation engines, and is now central to training language model agents through reinforcement learning from human feedback and from verifiable rewards such as passing tests. This lesson covers the RL problem, exploration versus exploitation, temporal-difference learning and Q-learning, deep reinforcement learning and policy gradients, and how RL is used to train modern agents, along with its characteristic failure modes.

The reinforcement learning problem

In RL, an agent interacts with an environment over a sequence of time steps. At each step it observes a state, takes an action, receives a reward and moves to a new state. Its goal is to learn a policy that maximizes the expected discounted return, the sum of future rewards, exactly as in an MDP, but without knowing the transition probabilities or rewards in advance. The agent must discover which actions lead to reward, often when rewards are delayed: a move early in a game may determine the outcome many moves later, and the agent must work out which earlier decisions deserve credit, the credit assignment problem.

RL methods are often divided into model-based methods, which learn a model of the environment and plan with it, and model-free methods, which learn values or policies directly from experience. Model-free methods are simpler and widely used; model-based methods can be far more sample-efficient, which matters when experience is expensive, as with real robots.

Exploration and exploitation

A learning agent faces a fundamental dilemma: exploit what it already knows to collect reward, or explore unfamiliar actions that might be better. Too little exploration risks settling on a mediocre policy forever; too much wastes time on poor actions. The simplest strategy, epsilon-greedy, takes the best-known action most of the time and a random action with small probability \(\varepsilon\). Other strategies include optimistic initial values, which make untried actions look attractive, upper confidence bounds, which favour actions whose value is uncertain, and Thompson sampling, which samples actions according to the probability that each is best. The same dilemma appears in business (trying new suppliers versus sticking with a good one), in A/B testing, and in agents deciding whether to try a new approach to a task.

Temporal-difference learning and Q-learning

Temporal-difference (TD) methods learn from each step by comparing their prediction with a slightly better-informed estimate. Q-learning, introduced by Watkins in 1989, learns an action-value function \(Q(s, a)\), the expected return of taking action \(a\) in state \(s\) and acting optimally afterwards. After each transition from \(s\) to \(s'\) with reward \(r\), it updates:

The term in brackets is the TD error, the difference between the new estimate and the old one; the learning rate \(\alpha\) controls how far to move. Q-learning is off-policy: it learns the value of the optimal policy even while following an exploratory one, and it is guaranteed to converge to the optimal values in finite problems if every state–action pair is visited often enough and the learning rate decreases appropriately. SARSA is the on-policy alternative, which learns the value of the policy actually being followed, including its exploration.

import numpy as np
# Q-learning on a cliff-walking task: learn from trial and error
rng = np.random.default_rng(5)
rows, cols = 4, 8
start, goal = (3, 0), (3, 7)
cliff = {(3, c) for c in range(1, 7)}
moves = [(-1, 0), (1, 0), (0, -1), (0, 1)]
def env_step(s, a):
    r = min(max(s[0] + moves[a][0], 0), rows - 1); c = min(max(s[1] + moves[a][1], 0), cols - 1)
    if (r, c) in cliff:
        return start, -100, False
    return (r, c), -1, (r, c) == goal
def train(eps, episodes=600, alpha=0.5, gamma=1.0):
    Q = np.zeros((rows, cols, 4)); returns = []
    for ep in range(episodes):
        s, total, done = start, 0, False
        for t in range(200):
            a = rng.integers(4) if rng.random() < eps else int(np.argmax(Q[s]))
            s2, r, done = env_step(s, a); total += r
            Q[s][a] += alpha * (r + gamma * np.max(Q[s2]) * (not done) - Q[s][a])
            s = s2
            if done:
                break
        returns.append(total)
    return Q, np.mean(returns[-100:])
for eps in (0.05, 0.1, 0.3):
    Q, avg = train(eps)
    s, path = start, 0
    for t in range(30):
        s, r, done = env_step(s, int(np.argmax(Q[s]))); path += 1
        if done:
            break
    print(f"epsilon {eps:4.2f}: mean return last 100 episodes {avg:7.1f}, greedy path length {path}")

epsilon 0.05: mean return last 100 episodes   -16.5, greedy path length 9
epsilon 0.10: mean return last 100 episodes   -26.8, greedy path length 9
epsilon 0.30: mean return last 100 episodes  -106.1, greedy path length 9

In this classic task, the agent must walk from the bottom left to the bottom right of a grid; stepping into the cliff along the bottom costs −100 and sends it back to the start, and every step costs −1. With all three exploration rates, the learned greedy policy takes the optimal 9-step path along the cliff edge. But the returns collected during learning differ sharply: with \(\varepsilon = 0.3\), random exploratory steps along the edge frequently fall into the cliff, giving an average return of −106, while \(\varepsilon = 0.05\) averages −16.5. Exploration has a real cost while learning, a serious concern when an agent learns in the real world, where exploration can break equipment or upset customers. Safe exploration, learning in simulation first, and learning from logged data (offline RL) are responses to this problem.

Deep reinforcement learning

Tables of Q-values work only for small problems. Deep reinforcement learning uses neural networks to approximate values or policies over huge state spaces, such as game screens or robot sensor readings. DeepMind's Deep Q-Network (DQN) learned to play dozens of Atari games from pixels in 2015, using experience replay (training on a buffer of past transitions) and a slowly updated target network to stabilize learning. Policy gradient methods instead adjust a parameterized policy directly in the direction that increases expected return, using sampled trajectories; actor–critic methods combine a policy (the actor) with a learned value function (the critic) that reduces variance. Proximal Policy Optimization (PPO) constrains each update to stay close to the previous policy, making training stable, and became the workhorse algorithm for many RL applications, including fine-tuning language models. AlphaGo, AlphaZero and MuZero combined deep RL with search, and RL now controls data centre cooling, chip floor planning and robotic manipulation in research and industry.

Rewards, shaping and reward hacking

An RL agent optimizes the reward it is given, not the intention behind it. Sparse rewards, given only at the end of a long task, make learning slow; reward shaping adds intermediate rewards to guide the agent, but poorly designed shaping can change what is optimal. Reward hacking (or specification gaming) occurs when an agent finds an unintended way to collect reward: a boat-racing agent that circled endlessly collecting points instead of finishing the race, a simulated robot that learned to exploit physics bugs, or a coding agent that modifies tests so they pass instead of fixing the code. Such behaviour is not malicious; it is the agent doing exactly what the reward specifies. Defences include careful reward design, testing for exploits, combining multiple signals, human oversight, and restricting the actions available to the agent.

Reinforcement learning for language model agents

Language models are pre-trained to predict text, then fine-tuned to follow instructions. Reinforcement learning from human feedback (RLHF) trains a reward model on human comparisons between responses and then optimizes the language model against it, typically with PPO or simpler methods such as direct preference optimization (DPO), which learns from preferences without an explicit RL loop. Constitutional AI and RL from AI feedback use model-generated judgements guided by written principles. For agents, reinforcement learning with verifiable rewards trains models on tasks with checkable outcomes, such as passing unit tests, reaching correct answers, or completing web tasks, rewarding successful multi-step trajectories. This has produced large gains in coding and reasoning agents, and it inherits RL's classic risks: reward hacking, such as special-casing tests, and behaviours that score well but do not match what users want. Evaluations that check for such shortcuts are now an important part of agent development (Lesson 12).

Imitation learning and learning from demonstrations

Reinforcement learning from scratch can be slow and risky, so agents often start by imitating experts. Behaviour cloning trains a policy with supervised learning on demonstrations of states and expert actions; it is simple but suffers when the agent drifts into situations the expert never encountered, where small errors compound. Methods such as DAgger collect expert corrections in those situations, and inverse reinforcement learning infers the reward an expert seems to optimize. Language model pre-training and supervised fine-tuning on examples of good behaviour are forms of imitation, after which reinforcement learning refines the behaviour toward outcomes. This combination, imitation first and reinforcement learning second, is now standard for training capable agents, from robots (AI-450) to coding assistants.

Evaluating learned policies

Evaluating an RL agent requires care. Training returns include exploration and change as the policy improves, so the final policy should be evaluated separately, without exploration, over many episodes and random seeds, because RL results vary considerably between training runs. Policies should also be tested on situations outside the training distribution, since agents often overfit to quirks of their training environment, and checked for unintended strategies by watching what they actually do.

Where reinforcement learning fits

Reinforcement learning is the right tool when good behaviour can be recognized through a reward signal but not easily specified step by step, when many trials are affordable, and when the environment can be simulated or safely explored. When an accurate model exists, planning may be simpler; when expert demonstrations exist, imitation provides a strong start; and when mistakes are costly, offline learning and careful constraints come first.

Reinforcement learning also connects to how people and organizations learn. Experimenting, measuring results, and adjusting is the same loop, with the same risks of optimizing a misleading metric. Keeping the true goal in view, and checking that improvements in the measured reward reflect improvements in what really matters, is good practice for both learning agents and the teams that build them.