Model-based and offline reinforcement learning

Most of the deep RL methods in this course learn from enormous amounts of interaction: millions of game frames or simulated robot steps. In many real problems that is impossible. A hospital cannot try random treatments, a self-driving car cannot crash to learn, and an industrial plant cannot be run badly for weeks. Two families of methods reduce the need for online interaction. Model-based RL learns a model of the environment and uses it to generate experience or to plan, extracting more from each real step. Offline (batch) RL learns a policy entirely from a fixed data set collected earlier, with no further interaction at all. Lesson 7 introduced both briefly; this lesson develops them, including the problems that make them hard, compounding model error and distribution shift, and the conservative methods that address those problems.

Learning models

A learned dynamics model predicts the next state (or a distribution over next states) and reward from the current state and action. Choices include:

• Deterministic or probabilistic: probabilistic models represent the noise in the environment and the model's own uncertainty, which matters because a policy optimized against a confident but wrong model will exploit its errors.
• Ensembles: training several models on bootstrapped data and measuring their disagreement gives a practical estimate of epistemic uncertainty (AI-320, Lesson 12). Probabilistic ensembles are the default in many model-based algorithms.
• Observation space or latent space: predicting future images pixel by pixel is expensive and wasteful; latent models such as those in Dreamer and MuZero (Lesson 6) predict compact learned representations instead.

Models are trained by supervised learning on collected transitions, which is far more data-efficient than learning values or policies, because every transition provides a full prediction target.

Compounding error

A model that is accurate for one step can be badly wrong after many, because each prediction is fed back as the input to the next, and small errors accumulate:

import numpy as np

rng = np.random.default_rng(10)
# True dynamics: an undamped oscillator rotating 0.10 rad per step. The learned model is 3% too fast.
rot = lambda w: np.array([[np.cos(w), np.sin(w)], [-np.sin(w), np.cos(w)]])
A_true, A_model = rot(0.10), rot(0.103)
x0 = np.array([1.0, 0.0])
for H in (1, 5, 20, 50, 100):
    xt, xm = x0.copy(), x0.copy()
    for _ in range(H):
        xt = A_true @ xt; xm = A_model @ xm
    print(f"rollout horizon {H:3d}: model error {np.linalg.norm(xm - xt):.3f} (true state size {np.linalg.norm(xt):.3f})")

rollout horizon   1: model error 0.003 (true state size 1.000)
rollout horizon   5: model error 0.015 (true state size 1.000)
rollout horizon  20: model error 0.060 (true state size 1.000)
rollout horizon  50: model error 0.150 (true state size 1.000)
rollout horizon 100: model error 0.299 (true state size 1.000)

A model whose frequency is wrong by only 3% is nearly perfect for one step, but its predicted state drifts away linearly, reaching 30% of the state's size after 100 steps. With nonlinear dynamics, errors often grow faster, and rollouts can enter regions where the model was never trained and makes arbitrary predictions. Worse, a policy optimized against the model will seek out states where the model is wrongly optimistic, a form of reward hacking against the model.

Practical model-based algorithms therefore manage how the model is used:

• Short rollouts: model-based policy optimization (MBPO) starts many short model rollouts (often 1 to 5 steps) from real states in the replay buffer and adds the simulated transitions to the data for a model-free learner such as SAC. Short rollouts limit compounding while still multiplying the data.
• Uncertainty penalties and termination: rollouts are stopped, or rewards penalized, where ensemble members disagree.
• Re-planning: model predictive control (MPC) plans a sequence of actions over a short horizon with the model, executes only the first action, observes the real outcome, and plans again, so model errors are corrected at every step.
• Value functions at the horizon: planning a few steps with the model and then using a learned value function for the rest (model-based value expansion) combines the strengths of both.

Model-based algorithms

• PILCO used Gaussian process dynamics models (AI-320, Lesson 6) with full uncertainty propagation and learned to swing up and balance a real cart-pole in about 20 seconds of experience, a dramatic demonstration of data efficiency, though it scaled poorly to high dimensions.
• PETS (probabilistic ensembles with trajectory sampling) combined probabilistic ensembles with MPC using sampled action sequences, matching model-free performance on simulated robots with orders of magnitude less data.
• MBPO combined short model rollouts with soft actor-critic.
• Dreamer learns a recurrent latent world model from images and trains an actor and critic entirely on imagined latent trajectories. DreamerV3 used a single set of hyperparameters across many domains and was the first algorithm to collect diamonds in Minecraft from scratch without human data.
• MuZero (Lesson 6) plans with MCTS in a latent model trained only to predict rewards, values and policies.

Model-based methods shine when data are expensive and the dynamics are learnable; model-free methods remain simpler and can achieve higher final performance when data are cheap. Many current systems blend the two.

Offline reinforcement learning

In offline RL, the agent receives a fixed data set of transitions \(\mathcal{D} = \{(s, a, r, s')\}\), collected by some behaviour policy (people, an existing controller, past versions of a system), and must produce the best possible policy without further interaction. This setting is attractive because large logs already exist in healthcare, recommendation, robotics, energy systems and industrial control, and because no unsafe exploration is needed.

Naively applying an off-policy algorithm such as Q-learning to offline data usually fails badly. The cause is distribution shift: the learned policy chooses actions that the behaviour policy rarely or never took, and the Q-function's estimates for those actions are pure extrapolation. The maximization in the Bellman target, \(\max_{a'} Q(s', a')\), actively selects whichever unseen action has the most overestimated value, and errors propagate through bootstrapping. Online, the agent would try those actions and discover its mistake; offline, it never can.

import numpy as np

rng = np.random.default_rng(9)
# A one-step decision from 5 states with 4 actions (contextual-bandit style offline problem).
S, A = 50, 4
true_r = rng.normal(0, 1, (S, A))
behaviour = np.full((S, A), 0.03); behaviour[np.arange(S), np.argmax(true_r + rng.normal(0, 0.5, (S, A)), 1)] = 0.91
data_s = rng.integers(0, S, 1000)
data_a = np.array([rng.choice(A, p=behaviour[s] / behaviour[s].sum()) for s in data_s])
data_r = true_r[data_s, data_a] + rng.normal(0, 2.0, 1000)       # noisy rewards

counts = np.zeros((S, A)); sums = np.zeros((S, A))
np.add.at(counts, (data_s, data_a), 1); np.add.at(sums, (data_s, data_a), data_r)
mean = np.where(counts > 0, sums / np.maximum(counts, 1), 0.0)

def value(policy):
    return np.mean(true_r[np.arange(S), policy])

naive = np.argmax(np.where(counts > 0, mean, 0.0) + np.where(counts > 0, 0, 3.0 * rng.normal(0, 1, (S, A))), 1)   # unseen actions get arbitrary estimates
greedy_seen = np.argmax(np.where(counts > 0, mean, -np.inf), 1)
pessimistic = np.argmax(np.where(counts > 0, mean - 2.0 / np.sqrt(np.maximum(counts, 1)), -np.inf), 1)
bc = np.argmax(counts, 1)                                         # behaviour cloning: copy the most common action
print(f"{int((counts == 0).sum())} of {S * A} state-action pairs never appear in the data; {int(((counts > 0) & (counts <= 2)).sum())} appear only once or twice")
for name, pol in (("naive, unseen actions guessed", naive), ("greedy on seen actions", greedy_seen),
                  ("pessimistic (mean - 2/sqrt(n))", pessimistic), ("behaviour cloning", bc)):
    print(f"{name:31s}: true value {value(pol):+.3f}")
print(f"{"optimal":31s}: true value {np.mean(true_r.max(1)):+.3f}")

93 of 200 state-action pairs never appear in the data; 56 appear only once or twice
naive, unseen actions guessed  : true value +0.435
greedy on seen actions         : true value +1.018
pessimistic (mean - 2/sqrt(n)) : true value +1.071
behaviour cloning              : true value +1.077
optimal                        : true value +1.143

Letting unseen actions take arbitrary values gives a policy far worse than the behaviour policy. Restricting the choice to observed actions helps, but some of those were seen only once or twice, and their noisy averages fool the greedy choice. Subtracting an uncertainty penalty that shrinks with the number of observations improves on greedy selection. Here, behaviour cloning does about as well as any method, because the behaviour policy was already good; offline RL earns its keep when the data contain a mix of good and poor behaviour from which a better policy can be stitched together.

Conservative offline methods

Offline RL algorithms share one principle: stay close to the data, or be pessimistic about actions the data do not support. Main approaches:

• Policy constraints: keep the learned policy close to the behaviour policy, by restricting it to actions a generative model of the behaviour finds likely (BCQ) or by adding a divergence or behaviour cloning penalty (TD3+BC, which simply adds a behaviour cloning term to the TD3 actor loss and is a strong, simple baseline).
• Value regularization: conservative Q-learning (CQL) adds a term that pushes down Q-values of actions sampled from the current policy and pushes up Q-values of actions in the data, so that the learned Q-function lower-bounds the true value for unsupported actions.
• In-sample learning: implicit Q-learning (IQL) never queries Q-values of unseen actions at all, using expectile regression over actions in the data set to approximate the maximum, then extracting a policy by advantage-weighted regression.
• Model-based offline RL: learn a model from the data and penalize simulated rewards by model uncertainty (MOPO, MOReL), so the policy avoids regions where the model cannot be trusted.
• Sequence modelling: the Decision Transformer treats offline RL as conditional sequence prediction, predicting actions given desired future returns, states and past actions, trained with ordinary supervised learning.

Off-policy evaluation for sequential decisions

Before deploying an offline-learned policy, its value must be estimated from data, extending the bandit methods of Lesson 2 to sequences. Importance sampling multiplies the probability ratios along a trajectory, \(\rho_{0:T} = \prod_t \pi(a_t \mid s_t)/\mu(a_t \mid s_t)\), which is unbiased but has variance that grows exponentially with the horizon; per-decision and weighted importance sampling reduce the variance. Fitted Q evaluation learns the target policy's Q-function from the data by repeated regression and averages it over initial states; it has low variance but inherits the extrapolation problems of offline learning. Doubly robust estimators combine the two. In practice, offline evaluation is used to rank candidate policies and screen out bad ones, followed by cautious online testing with guardrails.

Where these methods are used

• Robotics: model-based methods and offline RL from teleoperation logs reduce costly real-world trials; large robot data sets support offline pre-training followed by brief online fine-tuning.
• Healthcare: offline RL has been studied for treatment strategies such as sepsis management and dosing, with off-policy evaluation and clinical review, though the field emphasizes caution because confounding in observational data (AI-320, Lesson 4) can mislead.
• Recommendation and advertising: logs of past interactions train policies for long-term engagement rather than immediate clicks.
• Energy and industrial control: data centre cooling control, building HVAC and process plants, where historical operating data are plentiful and exploration is risky; learned policies typically run with safety constraints and human oversight.
• Language models: learning from logged human preferences and feedback (Lesson 12) is a form of offline RL.