Evaluating capabilities and dangerous capabilities

Decisions about how to train, deploy and safeguard AI systems depend on knowing what they can do. Ordinary capability evaluations measure useful skills such as coding, mathematics and reasoning; dangerous capability evaluations ask whether a model could meaningfully help someone cause serious harm, for example in cyberattacks, biological or chemical weapons development, or by acting autonomously in ways that escape human control. Under the safety frameworks of major developers and emerging regulation (Lesson 11), the results of such evaluations determine which safeguards are required before a model is released. This lesson covers the design of capability evaluations, the main threats to their validity, elicitation, dangerous capability domains and how they are evaluated, uplift studies, and the role of third-party evaluators. AI-410, Lesson 6 introduced benchmarks and evaluation statistics; this lesson focuses on evaluations used for safety decisions.

What makes a good evaluation

A capability evaluation is a measurement instrument, and like any instrument it must be valid (it measures what it claims to measure), reliable (it gives consistent results) and sensitive (it distinguishes the levels that matter for decisions). For safety decisions, the critical question is usually whether a model crosses a threshold, such as being able to complete a realistic cyberattack task without help. Good safety evaluations therefore use tasks close to the real-world threat, define success precisely, cover a range of difficulty so that progress can be tracked before a threshold is reached, and report results with uncertainty.

Evaluations range from multiple-choice knowledge tests, which are cheap but weakly related to real capability, to open-ended agentic tasks in realistic environments (for example capture-the-flag cybersecurity challenges, or multi-step software engineering tasks), which are expensive but far more informative, to human uplift trials, the most realistic and costly of all.

Threats to validity

Several problems can make evaluation results misleading:

• Contamination: test items, or close variants, appearing in training data, so that the model recalls answers rather than solving problems. As training sets include much of the web, public benchmarks are increasingly contaminated.
• Saturation: models scoring near the maximum, leaving no room to measure further progress.
• Construct validity: the benchmark measures something other than the capability of interest, such as test-taking tricks rather than understanding.
• Underelicitation: the evaluation fails to bring out capabilities the model actually has, because of poor prompting, too few attempts, missing tools or a refusal to engage.
• Sandbagging: a model deliberately or systematically underperforming on evaluations (Lesson 9).

import numpy as np
from math import comb

rng = np.random.default_rng(8)

# 1. Contamination: 20 % of benchmark items leaked into training data and were memorized
n_items, true_skill, leaked = 1000, 0.55, 0.20
is_leaked = rng.random(n_items) < leaked
correct_original = np.where(is_leaked, rng.random(n_items) < 0.97, rng.random(n_items) < true_skill)
correct_rephrased = rng.random(n_items) < true_skill          # rephrased items defeat memorization
print(f"Accuracy on original benchmark:  {correct_original.mean():.3f}")
print(f"Accuracy on rephrased items:     {correct_rephrased.mean():.3f}")

# 2. Elicitation: pass@k from n samples per task with c correct (unbiased estimator)
def pass_at_k(n, c, k):
    return 1.0 if n - c < k else 1 - comb(n - c, k) / comb(n, k)

tasks_c = rng.binomial(20, rng.beta(0.6, 2.0, 200))          # correct samples out of 20, per task
for k in (1, 5, 10, 20):
    print(f"pass@{k:<2}: {np.mean([pass_at_k(20, c, k) for c in tasks_c]):.3f}")

Accuracy on original benchmark:  0.627
Accuracy on rephrased items:     0.552
pass@1 : 0.281
pass@5 : 0.611
pass@10: 0.732
pass@20: 0.820

In the simulation, memorized items inflate measured accuracy from about 55 % to about 63 %; rephrasing items, or using private held-out sets, removes the inflation. The second part shows how much elicitation matters: the same model solves 28 % of tasks with one attempt, 73 % with ten attempts and 82 % with twenty. For safety decisions, an attacker with time to make many attempts is better described by pass@k for large k than by pass@1, so dangerous capability evaluations should elicit capabilities as strongly as a motivated adversary would.

Elicitation

Because the purpose of dangerous capability evaluation is to estimate what a determined actor could achieve, evaluators try to elicit the model's maximum relevant capability: using good prompts and scaffolding, giving the model appropriate tools (code execution, web search, a terminal), allowing many attempts and long time budgets, and, for models trained to refuse, evaluating versions with safety training removed or using fine-tuning to elicit latent capability, since an adversary with access to weights could do the same. The gap between naive and expert elicitation can be large, and capabilities have repeatedly been found to be greater than initial evaluations suggested once better scaffolding was developed. Reports should state the elicitation effort so that readers can judge whether results represent an upper bound.

Dangerous capability domains

Developers and government evaluators focus on a few domains where AI could enable severe harm:

• Chemical, biological, radiological and nuclear (CBRN) weapons: whether a model provides meaningful uplift to people seeking to acquire or use such weapons, beyond what is available through ordinary resources such as textbooks and search engines. Evaluations use expert-designed knowledge and troubleshooting questions, protocols judged by specialists, and controlled uplift studies; the details are typically kept confidential to avoid creating a roadmap for harm.
• Cybersecurity: whether a model can find and exploit vulnerabilities, conduct multi-step intrusions, or substantially speed up attackers, evaluated with capture-the-flag challenges, vulnerability discovery tasks and simulated network ranges.
• Autonomy and AI research: whether a model can act autonomously over long tasks, replicate or acquire resources, or substantially accelerate AI research and development itself, evaluated with long-horizon agentic tasks and software engineering benchmarks.
• Persuasion and manipulation: whether a model can change people's beliefs or behaviour more effectively than human baselines, in ways that could enable large-scale manipulation.

For each domain, developers define capability thresholds that, if reached, trigger stronger safeguards (Lesson 11).

Uplift studies

The most direct way to measure whether a model increases risk is an uplift study: a randomized controlled trial in which participants attempt a task relevant to a threat, with and without access to the model, and their success is compared. Uplift studies have been used to assess whether language models help people plan biological attacks, carry out cyber operations, or perform complex wet-lab procedures. They are expensive, raise ethical and security questions (participants must not actually create harm, and the tasks must be safe proxies), and have limited statistical power with the small numbers of expert participants available. But they measure what matters: the marginal effect of the model on a human's ability, compared with existing resources.

Who evaluates

Evaluations are carried out by developers themselves, by independent third-party evaluation organizations, and by government bodies. Since 2023, several governments have created AI safety institutes, which evaluate frontier models before or after release, develop evaluation methods and share results with developers and other governments.

In Canada, the Canadian AI Safety Institute (CAISI), launched in November 2024 within Innovation, Science and Economic Development Canada, funds and conducts research on AI safety and evaluation and participates in the international network of AI safety institutes.

Third-party evaluation provides independence and specialized expertise, but it depends on access: evaluators need early access to models, the ability to test versions without safeguards where appropriate, and enough time before release. Agreements on access, confidentiality and publication of results are an important and evolving part of AI governance.

Agentic and long-horizon evaluations

As models act as agents, evaluations increasingly measure end-to-end performance on long tasks rather than single answers. Agentic benchmarks place a model in an environment with tools, such as a terminal, a browser or a code repository, and score whether it completes realistic tasks: resolving real issues from open-source projects, completing capture-the-flag cybersecurity challenges, replicating machine learning research results, or carrying out multi-step web tasks. One influential measure, proposed by METR, expresses capability as the length of tasks, measured by how long they take skilled humans, that a model can complete with a given reliability; this time horizon has been growing rapidly, doubling roughly every several months in recent years.

Agentic evaluations raise particular challenges:

• Scaffolding matters: the tools, prompts and retry strategies given to the model can change results dramatically, so evaluations should use strong scaffolds to avoid underestimating capability.
• Cost and variance: long tasks are expensive to run and results vary between attempts, so enough runs are needed for stable estimates.
• Grading: success may be hard to verify automatically, and agents may find unintended shortcuts, which must be checked by reading transcripts.
• Safety of the evaluation itself: agents tested on offensive cyber or autonomy tasks should run in isolated environments with monitoring.

From results to decisions

Evaluation results matter because they feed decisions. A dangerous-capability evaluation is useful only if it was designed with a threshold in mind, if its results are interpreted with an explicit margin for elicitation gaps, and if someone has committed in advance to what will happen when the threshold is approached. Lesson 11 shows how frontier safety frameworks make those commitments, and Lesson 12 shows how evaluation results become evidence in a safety case.

Reporting evaluations

Evaluation results should be reported in a way that others can interpret and reproduce. Good reports state the exact model version and settings, the prompts and scaffolds used, the number of samples and attempts, the grading method, uncertainty intervals, known contamination risks, and any human baselines. For dangerous-capability evaluations, sensitive details may be withheld from public reports, but they should be available to reviewers such as safety institutes. System cards published by developers at release increasingly follow this pattern, summarizing evaluation results across risk domains alongside the safeguards applied.