AI Safety, Alignment and Evaluation
Making AI systems do what we intend: reward models, scalable oversight, robustness, red teaming, interpretability, capability and alignment evaluations, monitoring and control, governance and safety cases.
AI-430 Year 4 3 credits 12 lessons · 4 h 57 min
Offered by HSM University · Artificial Intelligence
This is the American version of this course. See the Canadian version.
What you'll learn
- Explain why alignment is hard using specification gaming, Goodhart's law, goal misgeneralization and the risk landscape
- Describe reward modeling, RLHF and its alternatives, and measure and mitigate reward overoptimization
- Compare scalable oversight approaches, including debate, recursive decomposition and weak-to-strong generalization
- Evaluate robustness to adversarial examples, distribution shift, data poisoning and backdoors
- Plan red-team campaigns against jailbreaks and prompt injection and report attack success with confidence intervals
- Apply attribution methods, linear probes with control tasks, feature visualization and activation steering
- Explain mechanistic interpretability, including activation patching, superposition, sparse autoencoders and circuits
- Design capability and dangerous-capability evaluations with strong elicitation, contamination checks and pass@k
- Evaluate honesty, calibration, sycophancy and propensities, and account for evaluation awareness and sandbagging
- Design monitoring and AI control protocols and assess them for safety and usefulness against a red team
- Interpret frontier safety frameworks, compute thresholds, AI laws and the role of safety institutes
- Build and review a safety case that combines inability, control and trustworthiness evidence
Course content
12 lessons · 4 h 57 min total · final exam
- 1 Why alignment is hard 24 min · quiz
- 2 Reward models and overoptimization 25 min · quiz
- 3 Scalable oversight 25 min · quiz
- 4 Robustness and adversarial examples 25 min · quiz
- 5 Jailbreaks, prompt injection and red teaming 24 min · quiz
- 6 Interpretability: probes, attribution and features 25 min · quiz
- 7 Mechanistic interpretability 25 min · quiz
- 8 Evaluating capabilities and dangerous capabilities 24 min · quiz
- 9 Evaluating honesty, calibration and alignment 24 min · quiz
- 10 Monitoring and control 25 min · quiz
- 11 Safety frameworks, policy and governance 26 min · quiz
- 12 Safety cases and capstone project 25 min · quiz
- E Final exam 20 questions
Requirements
Recommended background. You can start any course at any time.
Description
Making AI systems do what we intend: reward models, scalable oversight, robustness, red teaming, interpretability, capability and alignment evaluations, monitoring and control, governance and safety cases.
Part of the Science stream, typically taken in year 4.