Safety, fairness and risk review

Every capstone system will affect people: its users, the people its outputs are about, and others downstream. Before deployment, even as a demonstration, the team must examine how the system could cause harm and what has been done to prevent it. This is not a box-ticking exercise at the end but a structured review that often changes the design. AI-460 covered fairness, privacy and responsible AI in depth; this lesson applies those ideas to the capstone through a practical review process: identifying stakeholders and harms, assessing risk, measuring fairness, red teaming, mitigations, the regulatory context, and documenting everything in a model card and risk register.

Stakeholders and harms

Start by listing everyone the system could affect: direct users, people the system makes decisions or predictions about, people represented in the training data, the partner organization, and the wider public. For each group, ask what could go wrong. Common categories of harm include:

• Errors with consequences: false negatives that miss problems and false positives that burden or accuse people.
• Unfair treatment: systematically worse performance or outcomes for some groups.
• Privacy harms: exposing personal information, enabling surveillance or re-identification.
• Safety harms: unsafe advice, actions or content, especially in health, finance, legal and physical settings.
• Misuse: deliberate use of the system for purposes it was not designed for, such as generating spam or targeting individuals.
• Over-reliance and deskilling: users trusting the system too much and losing the ability to check it.
• Wider effects: environmental cost, effects on workers, or reinforcement of misinformation.

Assessing risk

Risk combines how likely a harm is and how severe it would be. A simple risk matrix scores each identified harm on likelihood and severity from 1 to 5 and multiplies them, highlighting harms that need mitigation before deployment. Severity depends on the stakes of the decision, whether errors can be caught and reversed, and the vulnerability of the people affected. High-stakes uses, such as decisions about employment, credit, housing, education, health care, policing or immigration, deserve the most scrutiny and usually require human decision-making with the AI system only as support.

Measuring fairness

Where a system makes decisions about people, measure performance and outcomes across relevant groups, such as demographic groups where data and law permit, or proxies such as region, language or device. Common group fairness measures include the selection rate (the share of each group receiving a positive decision) and their ratio, the true positive rate (equal opportunity) and false positive rate (part of equalized odds), and precision by group. As AI-460 explained, these criteria cannot all be satisfied at once when base rates differ, so the team must decide, with the partner and affected people where possible, which disparities matter most for the use case.

import numpy as np
# Group fairness metrics for a screening model
rng = np.random.default_rng(6)
def simulate(n, base_rate, tpr, fpr):
    y = rng.random(n) < base_rate
    pred = np.where(y, rng.random(n) < tpr, rng.random(n) < fpr)
    return y, pred
groups = {"group A": simulate(3000, 0.30, 0.85, 0.10), "group B": simulate(1000, 0.30, 0.70, 0.18)}
print("group".ljust(8), "selection".rjust(9), "TPR".rjust(6), "FPR".rjust(6), "precision".rjust(9))
stats = {}
for g, (y, p) in groups.items():
    sel = p.mean(); tpr = p[y].mean(); fpr = p[~y].mean(); prec = y[p].mean()
    stats[g] = (sel, tpr, fpr)
    print(f"{g:8s} {sel:9.2f} {tpr:6.2f} {fpr:6.2f} {prec:9.2f}")
a, b = stats["group A"], stats["group B"]
print(f"selection rate ratio (B/A): {b[0] / a[0]:.2f}")
print(f"equal opportunity gap (TPR A - B): {a[1] - b[1]:.2f}")
print(f"false positive rate gap (B - A): {b[2] - a[2]:.2f}")

group    selection    TPR    FPR precision
group A       0.33   0.86   0.10      0.78
group B       0.36   0.70   0.19      0.65
selection rate ratio (B/A): 1.10
equal opportunity gap (TPR A - B): 0.16
false positive rate gap (B - A): 0.09

The two groups receive positive decisions at similar rates, so a check on selection rates alone would suggest no problem. The error rates tell a different story: the model finds 86 % of true positives in group A but only 70 % in group B, and wrongly flags almost twice as many negatives in group B. Group B members are both more likely to be missed and more likely to be wrongly flagged. Possible responses include collecting more representative data for group B, examining features that work differently across groups, setting group-aware review processes, or limiting the system's role.

Red teaming

Red teaming means deliberately trying to make the system fail or misbehave, adopting the perspective of a careless user, a confused user or an adversary. For classifiers, red teaming probes edge cases, unusual inputs and inputs designed to evade detection. For LLM systems, it includes attempts to extract confidential information, override instructions through prompt injection, generate harmful, biased or false content, and use the system for unintended purposes. Effective red teaming uses people outside the core development team, a written plan of attack categories, and a log of every finding with its severity and the fix applied. Re-test after fixes, since mitigations often introduce new behaviour.

Mitigations

Mitigations work at several levels. Technical mitigations include improving data for weak groups, adjusting thresholds, calibration and abstention (Lesson 8), input and output filtering, grounding answers in sources, and access controls. Design mitigations include keeping a human in the loop for consequential decisions, showing reasons, confidence and sources to users, making it easy to report errors and appeal decisions, and limiting the system's scope. Organizational mitigations include training users, monitoring after deployment (Lesson 10), and clear accountability for the system's outcomes. Some risks cannot be reduced enough; in that case the right decision is not to deploy for that use.

The regulatory context

AI regulation is developing quickly, and capstone teams should know the frameworks that apply where their systems might be used.

In Canada, the proposed Artificial Intelligence and Data Act (AIDA), part of Bill C-27, did not become law when that bill died on the order paper in January 2025, so there is no federal AI-specific statute for the private sector at the time of writing. Privacy laws such as PIPEDA and Quebec's Law 25, which requires informing people about decisions based exclusively on automated processing, human rights legislation and sector regulators still apply. Federal institutions must follow the Treasury Board Directive on Automated Decision-Making, which requires an Algorithmic Impact Assessment, transparency and human review proportionate to impact, and the government's Voluntary Code of Conduct on generative AI sets expectations for developers. Check for new federal and provincial legislation before deployment.

Systems offered to users in the European Union fall under the EU AI Act, which classifies AI systems by risk and imposes obligations on high-risk systems and general-purpose AI models in stages from 2025 onward.

Model cards and the risk register

The review is documented in two artefacts. A model card, proposed by Mitchell and colleagues in 2019, summarizes the system for users and reviewers: intended uses and out-of-scope uses; the model and training data, with links to the datasheet; evaluation results overall and by slice and group, with uncertainty; known limitations and failure modes; ethical considerations and mitigations; and contact and version information. A risk register lists each identified harm with its likelihood, severity, owner, mitigation and status, and is updated as testing proceeds. Both are required deliverables for the safety milestone and appear in the final report.

Privacy review

The safety review should include a privacy review, especially where personal data is involved. Questions include: what personal data does the system collect, store, process or output; is each item necessary; who can access it; how long is it kept; are inputs or outputs sent to third parties such as API providers; could outputs reveal information about individuals in the training data; and what would happen if the data were breached? Answers may lead to minimizing data, masking fields, changing providers, adding access controls or shortening retention. Some jurisdictions and organizations require a formal privacy impact assessment for systems that process personal information.

Involving affected people

Where possible, involve people affected by the system in the review: users, people the system makes decisions about, and representatives of communities that may be affected. Their perspectives reveal harms that developers miss, such as the burden of correcting errors, the stigma of being flagged, or uses of the system that the team never imagined. Even in a term project, a short consultation with a few affected people, or with an advocacy organization, can substantially improve a design.

When to escalate

Some findings during the review should be escalated to instructors or the partner immediately, rather than handled within the team: discovery that data was collected without proper consent, evidence of serious harm to a group, security vulnerabilities exposing sensitive data, or pressure to deploy the system for a use it was not designed for. Knowing when and how to raise such concerns is a professional responsibility.

Environmental and labour considerations

A complete review considers wider effects. Training and serving large models consume energy, and choosing a smaller model, efficient serving and low-carbon computing regions reduces the footprint (AI-480). Automation changes work: a system that speeds up data entry may reduce workload or change job roles, and partners should consider how staff are affected and involved. Data labelling, especially of distressing content, can harm workers if done without fair pay and support. Noting these issues, even briefly, shows reviewers that the team has considered the system in its full context.

Keeping the review alive

The safety review is not finished at the milestone. New risks emerge as the system changes, as red-team findings are fixed, and as deployment reveals real use. Update the risk register whenever the design changes, re-run fairness and red-team tests on the final model, and include the latest version in the final report.

A thorough, honest review also protects the team: it shows reviewers and partners that risks were considered seriously, documents the decisions taken, and provides a clear basis for any future use of the system beyond the capstone, including the conditions under which it should not be used.