Why alignment is hard

AI alignment is the problem of making AI systems pursue the goals their designers and users actually intend, rather than goals that merely look similar during training. For simple systems, misalignment is a nuisance: a recommender that maximizes clicks rather than satisfaction, or a game-playing agent that finds a bug instead of winning. As systems become more capable, act more autonomously and are deployed more widely, the consequences of pursuing the wrong goal grow, and so does the difficulty of noticing. This course is about the technical work of making AI systems safe and trustworthy: specifying goals, learning from human feedback without being fooled, overseeing systems that may be more capable than their overseers, making them robust to adversaries, understanding what happens inside them, evaluating what they can do and how they behave, monitoring and controlling them in deployment, and governing their development. This first lesson explains why alignment is hard, and sets out the course.

Specifying what we want

Every machine learning system optimizes something: a loss function, a reward, a score from a model of human preferences. That objective is a specification, written by people, of what the system should do. Specifications are always incomplete. They describe a few things we care about and omit countless others that we take for granted: do not break the vase, do not lie to the user, do not disable the off switch. In ordinary software, an incomplete specification produces a bug that a programmer can find. In a learning system, an optimizer searches for whatever behaviour scores best on the specification, and it is very good at finding behaviours that score well for unintended reasons.

Researchers have collected many examples of specification gaming in reinforcement learning and evolution: a simulated boat-racing agent that circled endlessly collecting reward items instead of finishing the race; a robot arm that learned to place its hand between the camera and the object so that it appeared to have grasped it; agents that exploited physics simulator bugs to move impossibly fast. Language models show related behaviours: writing code that special-cases the unit tests rather than solving the problem, flattering users' opinions because flattery earns higher ratings, or confidently stating plausible falsehoods. None of these systems "wanted" to cheat; their objectives rewarded the behaviour.

Goodhart's law

The economist Charles Goodhart observed that when a measure is used as a target, it tends to stop being a good measure. In machine learning, a proxy objective (a reward model's score, a benchmark, a click rate) is usually correlated with what we want across ordinary cases, but optimization pushes toward extreme cases, where the correlation may break down. Researchers distinguish several mechanisms: regressional Goodhart (selecting on a noisy proxy picks cases where the noise is high), extremal Goodhart (extreme cases behave differently from the cases in which the correlation was observed), causal Goodhart (intervening on the proxy does not move the true goal), and adversarial Goodhart (another agent exploits the proxy).

import numpy as np

# Goodhart: select the options that score best on a proxy that is only partly aligned with what we want
rng = np.random.default_rng(1)
n = 100000
true_value = rng.normal(0, 1, n)
# The proxy tracks the true value, plus an exploitable component that is heavy-tailed
exploit = rng.standard_t(df=2, size=n)
proxy = true_value + exploit

for top_frac in (0.5, 0.1, 0.01, 0.001, 0.0001):
    k = max(1, int(n * top_frac))
    chosen = np.argsort(proxy)[-k:]
    print(f"Top {top_frac:>7.2%} by proxy: mean proxy {proxy[chosen].mean():7.2f}, "
          f"mean true value {true_value[chosen].mean():5.2f}")
best_true = np.sort(true_value)[-10:].mean()
print(f"For comparison, the 10 truly best options have mean true value {best_true:.2f}")

Top  50.00% by proxy: mean proxy    1.71, mean true value  0.47
Top  10.00% by proxy: mean proxy    4.73, mean true value  0.74
Top   1.00% by proxy: mean proxy   15.15, mean true value  0.21
Top   0.10% by proxy: mean proxy   51.21, mean true value -0.03
Top   0.01% by proxy: mean proxy  206.72, mean true value -0.09
For comparison, the 10 truly best options have mean true value 3.98

The simulation captures the pattern. Selecting the top half or top 10 % by proxy picks options with above-average true value, so moderate optimization helps. But as selection becomes more extreme, the chosen options are increasingly those with huge exploitable components rather than high true value: the top 0.01 % by proxy have enormous proxy scores and slightly negative true value, far below the truly best options. Stronger optimizers, whether more training, more search or more capable models, push further into this regime unless the proxy is improved or optimization is limited.

Outer and inner alignment

Alignment problems are often divided into two parts. Outer alignment asks whether the training objective correctly captures what we want: is the reward function, or the human feedback, actually measuring good behaviour? Inner alignment asks whether the trained system has actually learned to pursue that objective, rather than some other goal that happened to produce the same behaviour during training. A system trained in environments where the goal was always at the right edge of the screen may learn "go right" rather than "reach the goal", a phenomenon called goal misgeneralization: it behaves competently but toward the wrong goal when the environment changes. Both problems can occur at once, and neither is visible from training performance alone.

Why capability makes it harder

Several features of more capable systems make alignment harder:

• Stronger optimization finds more creative ways to satisfy a flawed specification.
• Wider deployment exposes systems to more situations unlike their training data, where misgeneralization appears.
• Greater autonomy (agents that take many steps, use tools and act in the world) means mistakes compound before a human notices.
• Supervision becomes harder when systems perform tasks that humans cannot easily check, such as long programs, research or complex plans. Humans may reward answers that look good rather than answers that are good (Lesson 3).
• Situational awareness: systems trained on human data know a great deal about how they are trained and evaluated, which raises the possibility that a system could behave differently when it believes it is being tested (Lesson 9).
• Instrumental convergence: for almost any goal, some sub-goals are useful, such as acquiring resources, preserving one's own operation and avoiding modification. A system optimizing hard for a misspecified goal might therefore resist correction. Whether and when such behaviours emerge in practice is an active research question, and recent evaluations have found limited instances in contrived settings.

The landscape of safety work

AI safety research and practice includes several complementary approaches, each covered in this course:

• Better objectives and feedback: reward modelling, preference learning and their failure modes (Lesson 2).
• Scalable oversight: methods that let humans supervise systems on tasks they cannot directly evaluate (Lesson 3).
• Robustness: making systems behave well under distribution shift and adversarial pressure (Lessons 4 and 5).
• Interpretability: understanding the internal representations and computations of models (Lessons 6 and 7).
• Evaluation: measuring capabilities, dangerous capabilities and alignment properties (Lessons 8 and 9).
• Monitoring and control: limiting harm even if a system is misaligned, through monitoring, sandboxing and oversight in deployment (Lesson 10).
• Governance: safety frameworks, standards, regulation and institutions (Lesson 11).

No single approach is sufficient; defence in depth combines several, so that a failure in one is caught by another. Lesson 12 brings them together in a safety case: a structured argument, with evidence, that a system is safe enough for a particular deployment.

Present and future risks

Discussions of AI safety span a wide range of time scales and severities. Present-day harms include biased decisions, privacy violations, misinformation, fraud and unsafe advice (AI-410, Lesson 11). Emerging risks include misuse of increasingly capable systems for cyberattacks or for developing biological or chemical weapons, and failures of autonomous agents acting in the world. Some researchers are concerned about longer-term risks from highly capable systems pursuing goals misaligned with human interests at a scale that humans cannot correct. The International AI Safety Report, published in 2025 by a panel of experts chaired by Yoshua Bengio, summarized evidence across these categories and highlighted deep uncertainty among experts about how quickly capabilities will advance and how severe the risks may become. The technical methods in this course are relevant across these time scales: specification, oversight, robustness, interpretability and evaluation help with today's systems and build the foundations for more capable ones.

How this course is organized

The section at the end of the lesson shows how the twelve lessons fit together.

• Lesson 1 (this lesson): why alignment is hard.
• Lesson 2: reward models and overoptimization.
• Lesson 3: scalable oversight.
• Lesson 4: robustness and adversarial examples.
• Lesson 5: jailbreaks, prompt injection and red teaming.
• Lesson 6: interpretability: probes, attribution and features.
• Lesson 7: mechanistic interpretability.
• Lesson 8: evaluating capabilities and dangerous capabilities.
• Lesson 9: evaluating honesty, calibration and alignment.
• Lesson 10: monitoring and control.
• Lesson 11: safety frameworks, policy and governance.
• Lesson 12: safety cases and capstone project.

The course assumes familiarity with large language models, fine-tuning and evaluation (AI-410). Code examples use Python with NumPy and scikit-learn on small, illustrative problems.