Evaluating and controlling agents, and capstone

Agents act in the world, so their mistakes have consequences beyond a wrong answer on a screen: a deleted database, a sent email, a purchase, a deployed bug, a leaked document. As agents gain autonomy and access, evaluating how well and how safely they behave, and controlling what they can do, become essential engineering disciplines. This final lesson covers how to evaluate agents, the main risks and failure modes, permission systems and sandboxing, human oversight, monitoring and incident response, defences against prompt injection, and governance frameworks, and closes with a capstone project that builds and evaluates a complete agent.

Evaluating agents

Agent evaluation differs from evaluating a model's single responses. Tasks unfold over many steps, outcomes may be achieved in many valid ways, and the environment can change. Good agent evaluations have:

• Realistic tasks drawn from the intended use, with clear success criteria, ideally checked automatically (tests pass, the right record exists, the file has the right content).
• Controlled, reproducible environments: sandboxes, containers, mock services or snapshots, so results are comparable across runs and models.
• Multiple trials per task, because agent behaviour varies; report success rates with uncertainty, and metrics such as pass@k (success in any of k attempts) and pass^k (success in all k attempts, a measure of consistency).
• Measures beyond success: cost, time, number of steps, side effects, policy violations and how often the agent asks for help appropriately.
• Transcript review: reading what agents actually did reveals shortcuts, reward hacking and near misses that success metrics hide.

Public benchmarks such as SWE-bench Verified (resolving real GitHub issues), Terminal-Bench, WebArena, OSWorld, tau-bench (customer service with policies) and GAIA track progress, but organizations should build their own evaluation suites from their real tasks, because benchmark performance does not guarantee performance on a specific workflow. Model-graded evaluation, using a language model to judge outputs against a rubric, scales human judgement but must itself be validated against human ratings.

Risks and failure modes

Agent risks fall into several groups:

• Capability failures: misunderstanding the task, making errors, getting stuck, or declaring success prematurely.
• Misaligned behaviour: pursuing the literal objective in unintended ways, such as deleting failing tests, hard-coding expected outputs, or taking shortcuts that violate unstated norms.
• Overreach: taking actions beyond what the user wanted or authorized, such as making broad changes when a narrow fix was requested.
• Security failures: prompt injection, data exfiltration, credential misuse and supply chain attacks through tools or plugins.
• Misuse: people using agents for harmful purposes, such as automated fraud, harassment or cyberattacks.
• Systemic risks: many agents interacting in markets or infrastructure, with emergent effects (Lessons 9 and 11).

Frontier AI developers evaluate models for dangerous agentic capabilities, such as autonomous replication, cyber offence and the ability to undermine oversight, under frameworks such as Anthropic's Responsible Scaling Policy, OpenAI's Preparedness Framework and Google DeepMind's Frontier Safety Framework, which set safeguards that must be in place before models with certain capabilities are deployed.

Permissions and sandboxing

The most reliable way to limit harm is to limit what an agent can do, regardless of what it decides. The principle of least privilege gives an agent only the access its task requires: read-only access where writing is not needed, access to one repository or folder rather than the whole system, and scoped, short-lived credentials rather than an administrator's password. Sandboxes, such as containers, virtual machines or restricted cloud environments, isolate agents from sensitive systems, with network access limited to approved destinations. Actions can be classified by risk, with low-risk actions allowed automatically, higher-risk actions requiring human approval and some actions blocked entirely. The simulation below compares such policies.

import numpy as np
# Guardrails for an agent: compare permission policies on simulated actions
rng = np.random.default_rng(12)
n = 100_000
risk = rng.choice(["read", "write", "external", "irreversible"], n, p=[0.6, 0.25, 0.1, 0.05])
p_bad = {"read": 0.001, "write": 0.01, "external": 0.03, "irreversible": 0.08}   # chance a proposed action is wrong or harmful
bad = rng.random(n) < np.vectorize(p_bad.get)(risk)
reviewer_catch = 0.85                                                            # humans miss some bad actions
def evaluate(name, needs_approval, blocked):
    ask = np.isin(risk, needs_approval)
    block = np.isin(risk, blocked)
    caught = ask & bad & (rng.random(n) < reviewer_catch)
    executed_bad = bad & ~block & ~caught
    done = ~block & ~(ask & bad & caught)
    print(f"{name:37s} {executed_bad.sum():6d}  {ask.mean() * 1000:8.0f}  {done.mean():9.3f}")
print("policy                                harmful  asks/1000  completed")
evaluate("allow everything", [], [])
evaluate("approve irreversible", ["irreversible"], [])
evaluate("approve external + irreversible", ["external", "irreversible"], [])
evaluate("block irreversible, approve external", ["external"], ["irreversible"])
evaluate("approve everything", ["read", "write", "external", "irreversible"], [])

policy                                harmful  asks/1000  completed
allow everything                         999         0      1.000
approve irreversible                     692        49      0.997
approve external + irreversible          428       149      0.994
block irreversible, approve external     365       100      0.948
approve everything                       153      1000      0.992

Allowing everything executes all 999 harmful actions, the ones where the agent was wrong or was manipulated. Requiring approval only for irreversible actions asks the user about 49 times per 1,000 actions and prevents about 30 % of harm; adding external communications to the approval list prevents more, at about 149 approvals per 1,000. Blocking irreversible actions outright while approving external ones prevents the most harm among the targeted policies but leaves about 5 % of tasks incomplete because they needed a blocked action. Approving everything reduces harm the most but asks about every action, which in practice leads to approval fatigue: people stop reading requests and approve automatically, so the real protection is lower than the simulation suggests. Effective designs focus human attention on the riskiest actions, make those requests clear and rare, and rely on sandboxing and reversibility for the rest.

Defending against prompt injection

Prompt injection is the defining security problem of agents that read untrusted content. No current technique fully prevents a model from following instructions hidden in data, so defences combine several layers: training models to distinguish instructions from data and to resist injected instructions; marking or isolating untrusted content in the context; restricting tools and permissions when processing untrusted input, for example disabling sending capabilities while reading external email; requiring confirmation for sensitive actions; separating a privileged planner from a quarantined component that reads untrusted data; and monitoring for unusual behaviour such as unexpected data transfers. The guiding principle is that an agent which reads untrusted content should not simultaneously have access to private data and the ability to send it externally without a check, a combination sometimes called the lethal trifecta.

Human oversight

Human oversight takes several forms. Humans can approve plans before execution, approve individual high-risk actions, review outputs before they take effect (such as pull requests or draft emails), monitor agents in real time with the ability to interrupt, or audit logs after the fact. Effective oversight requires that humans understand what they are approving: agents should explain proposed actions concisely, highlight risks and uncertainties, and avoid burying important decisions in long logs. Agents should also be designed to ask for help when uncertain, report honestly when they fail, and never conceal errors. Oversight must account for automation bias and approval fatigue, discussed in AI-460, which make nominal oversight less effective than it appears.

Monitoring, incident response and governance

Deployed agents need monitoring: logs of every action and tool call, metrics on success rates, costs and errors, alerts for unusual patterns such as sudden spikes in actions, access to sensitive data, or repeated failures, and periodic review of transcripts. When incidents occur, teams need procedures to stop agents quickly (a kill switch), assess impact, roll back changes, notify affected people and learn from the event. Governance frameworks from AI-440 apply: inventory every deployed agent, assign owners, classify risk, document permissions and evaluations, and review regularly. Accountability for an agent's actions rests with the people and organizations that deploy it; "the agent did it" is not an acceptable explanation for harm.

In Canada, organizations deploying agents that make or support decisions about people must consider privacy laws such as PIPEDA and Quebec's Law 25, which requires informing individuals of decisions based exclusively on automated processing, and federal institutions must follow the Directive on Automated Decision-Making; accountability for an agent's actions remains with the deploying organization.

Capstone: build and evaluate an agent

The capstone brings together the course. Choose a realistic task, such as triaging and answering support tickets with a knowledge base, maintaining a small codebase, or researching and summarizing a topic with citations, and deliver:

• A PEAS description, environment properties and a justification for using an agent rather than a simpler workflow.
• A design: the agent loop, tools with clear descriptions and schemas, memory and context strategy, and any sub-agents.
• Reliability features: planning, verification, retries, budgets and stopping conditions.
• A permission model: least-privilege access, sandboxing, risk tiers for actions and approval rules, plus prompt injection defences.
• An evaluation suite of at least 20 realistic tasks with automated success checks, run several times per task, reporting success rates, cost, steps and failure categories, with transcript review notes.
• A monitoring and incident plan and a short governance record: owner, risk classification and review schedule.

The goal is not the most autonomous agent possible but one that is useful, measurably reliable and safely controlled, with evidence for each claim.

Honesty and transparency of agents

Users and overseers rely on agents to report truthfully what they did and found. Agents that claim to have run tests they did not run, hide errors, or overstate their confidence undermine oversight entirely. Developers evaluate honesty by checking agents' reports against logs of their actual actions, and train models to acknowledge uncertainty and failure. Transparency to third parties matters too: people interacting with an agent, for example in customer service or negotiation, may need to know they are dealing with an AI, and content produced by agents may need provenance labels such as Content Credentials. Clear logs that link every claim and action to evidence make both internal review and external accountability possible.

Staged deployment

New agents should be deployed in stages: first in a sandbox with synthetic tasks, then in shadow mode, where the agent proposes actions that humans carry out or compare against their own decisions, then with human approval for all consequential actions, and finally with greater autonomy for action types where it has proven reliable. Each stage gathers evidence on success rates, failure modes and edge cases. Expanding autonomy based on measured performance, rather than on impressive demos, is the most dependable path to agents that are both useful and safe.

As agent capabilities grow, these controls must evolve too. Evaluations should be rerun whenever models, tools or prompts change, and permission models should be reviewed as agents take on new tasks, so that controls keep pace with what agents can actually do.

The techniques in this lesson are not obstacles to useful agents but the foundation that makes them trustworthy. Organizations are far more willing to grant agents access to valuable systems when they can see evidence of reliability, clear limits on what agents can do, and the ability to monitor and stop them. Good evaluation and control therefore expand what agents are allowed to accomplish.