Governing AI: Canadian law, global rules and a capstone case

On the last night of the seminar, Riverton General's board meets to decide on Pathfinder, and it has asked Varga's class for a written analysis. Raúl will present it. Over eleven sessions, the class has looked at the system through every lens the course offers: its values, its use of patient data, its fairness, its explainability, its effect on nurses, who would be responsible if it failed and whether its objective captures what the hospital wants. What remains is the question of rules: what the law requires, what it should require, and how a public institution should govern a powerful tool when the law is unsettled. Varga splits the session into two parts: first the governance landscape, then the capstone.

Why governance is hard

Governing AI is difficult for reasons the course has already met. The Collingridge dilemma means rules made early may miss the real problems, and rules made late may be too late. The pacing problem, a related idea, is that technology changes faster than law can adapt. AI is a general-purpose technology used in almost every sector, so rules designed for one use may fit another badly. Much of the expertise is concentrated in the companies to be regulated, creating a risk of regulatory capture, in which regulators come to serve the interests of the industry. And there is a genuine trade-off: rules that are too loose allow harm, and rules that are too strict or uncertain may slow beneficial uses or push development to places with weaker protections.

Governance does not mean only legislation. It includes a range of tools, from soft to hard.

• Professional and corporate self-governance: codes of ethics, internal review boards, published principles and safety policies.
• Technical standards: detailed, voluntary specifications that can later be referenced by law, such as the ISO/IEC 42001 standard for AI management systems published in 2023 and the AI Risk Management Framework published by the U.S. National Institute of Standards and Technology in the same year.
• Procurement: governments and large institutions can require suppliers to meet conditions, such as testing, documentation and audit rights, as a condition of purchase.
• Existing law applied to AI: privacy, consumer protection, human rights, product liability, negligence, employment and sector rules for areas such as health, finance and transport.
• AI-specific law: statutes that regulate AI systems as such, usually by their level of risk.
• International agreements: principles and treaties among states.

The European Union's AI Act

The most comprehensive AI-specific law is the European Union's Artificial Intelligence Act, adopted in 2024 and in force since August 2024, with its obligations phased in over several years. It takes a risk-based approach. A small set of practices is prohibited outright, including social scoring by public authorities, manipulative techniques that cause significant harm, exploiting the vulnerabilities of particular groups, untargeted scraping of facial images to build recognition databases, emotion recognition in workplaces and schools, and, with narrow exceptions, real-time remote biometric identification in public spaces for law enforcement. These prohibitions have applied since February 2025. High-risk systems, such as AI used in hiring, credit, education, essential public services, law enforcement, migration and the safety components of regulated products, must meet requirements for risk management, data quality, documentation, human oversight, accuracy and robustness. Systems that interact with people or generate content have transparency duties. Providers of general-purpose AI models have duties of documentation and copyright compliance, with additional duties for models that pose systemic risk, applying since August 2025.

In 2026 the EU amended the Act through what it called a digital omnibus, intended to simplify compliance. The amendment, in force since July 2026, postponed the main obligations for high-risk systems: for stand-alone high-risk uses, from August 2026 to December 2027, and for AI embedded in regulated products, to August 2028. It also added a new prohibition on AI systems designed to generate non-consensual intimate imagery or child sexual abuse material. Supporters said the delay gave time for standards and guidance to be ready; critics said it weakened protections under industry pressure.

Because many companies sell into the European market, EU rules often shape products used elsewhere, a pattern the legal scholar Anu Bradford has called the Brussels effect. Whether that will happen with the AI Act, as it did to a degree with the General Data Protection Regulation, is an open question.

AI governance in Canada

In June 2022, the federal government introduced the Artificial Intelligence and Data Act as part of Bill C-27. It would have regulated 'high-impact' AI systems, requiring their developers and operators to assess and mitigate risks, keep records and publish plain-language descriptions, under the oversight of a commissioner. Critics said the bill left too much to future regulations, had been drafted without enough consultation and did not cover government uses. The bill was still in committee when Parliament was prorogued in January 2025, and it died. In the meantime, the government had published a Voluntary Code of Conduct on the responsible development and management of advanced generative AI systems in September 2023, which a number of companies signed.

After AIDA lapsed, the new government appointed Canada's first Minister of Artificial Intelligence and Digital Innovation in 2025. In June 2026 it released a national AI strategy, AI for All, which chose not to propose a comprehensive AI law. Instead, it promised targeted measures on particular problems, such as modernizing privacy and online safety law, protecting children, giving people legal tools against harmful deepfakes and protecting elections, alongside funding for the Canadian AI Safety Institute and a proposed certification program for trusted AI. AI in Canada is therefore governed mainly by existing law: federal and provincial privacy laws, including Quebec's Law 25; human rights codes; consumer protection; the Competition Act; sector regulators such as Health Canada, which regulates AI used in medical devices; and, for federal institutions, the Directive on Automated Decision-Making from lesson six. Ontario's Strengthening Cyber Security and Building Trust in the Public Sector Act, 2024, gave the province power to set rules for AI use by public sector bodies, which can include hospitals.

International frameworks

Several international instruments set out shared principles. The OECD AI Principles, adopted in 2019 and updated in 2024, were the first intergovernmental standard on AI and were endorsed by dozens of countries, including Canada and the United States; they call for AI that is innovative, trustworthy and respects human rights and democratic values. UNESCO's Recommendation on the Ethics of Artificial Intelligence, adopted by its member states in 2021, is the first global standard on AI ethics. The Council of Europe's Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law, opened for signature in 2024, is the first legally binding international treaty on AI; it was signed at its opening by the European Union, the United Kingdom, the United States and several other states. The G7's Hiroshima process produced a code of conduct for developers of advanced AI. And as lesson eleven described, a series of summits has addressed the safety of frontier models.

Philosophical questions about governance

Varga closes the first half of the session with questions that no law settles. Who should decide how powerful technologies are governed: elected legislatures, expert regulators, courts, companies, the people most affected, or some combination? Democratic legitimacy argues for legislatures and public participation; expertise argues for specialized regulators; the speed of change argues for flexible tools. How precautionary should governance be? A precautionary approach places the burden of proof on those who want to deploy a risky technology; an innovation-first approach places it on those who want to restrict it. Each can be defended, and the right answer may differ by domain: medicine has long required evidence of safety before use, while most software has not. And how should governance address power? Several of the course's cases, from surveillance to the distribution of productivity gains, show that the deepest questions about AI are often questions about who holds power and how it is checked.

The capstone: a full analysis of Pathfinder

Raúl presents the seminar's analysis to the board. It follows a structure the class built over the term, which can be applied to any AI system.

# The seminar's rubric for Pathfinder: each concern rated 0 (addressed) to 3 (serious, unaddressed)
concerns = {
    "benefit is real and measured against current practice": 1,
    "patient data used within the norms of care": 1,
    "errors not concentrated on any group": 2,
    "staff can see why a patient was ranked": 2,
    "nurses can override and have time to": 2,
    "clear accountable owner and liability": 1,
    "objective matches the real goal (urgency, not past admissions)": 2,
    "monitoring and power to suspend": 1,
}
total = sum(concerns.values())
print("total concern score:", total, "of", 3 * len(concerns))
for k, v in concerns.items():
    if v >= 2:
        print("must fix before use:", k)

total concern score: 12 of 24
must fix before use: errors not concentrated on any group
must fix before use: staff can see why a patient was ranked
must fix before use: nurses can override and have time to
must fix before use: objective matches the real goal (urgency, not past admissions)

• Purpose and alternatives. Pathfinder is meant to get urgent patients seen sooner. The alternatives include more triage staff, better triage protocols and using the model only as a second opinion. The test results suggest a real benefit, but they must be compared with the current process measured the same way.
• Values and stakeholders. Patients, especially those the model may under-rank; nurses, whose judgment and workload are affected; the hospital; Brightline; and the public, which funds the hospital and expects fair care.
• Privacy. The training data must stay within the norms of clinical care: used to improve emergency care, not sold or used for other products, ideally kept inside the hospital's systems, and disclosed to patients.
• Fairness. Brightline must report error rates by age, sex and, where data allow, ethnicity and other characteristics, with particular attention to under-ranking of urgent patients. The finding that the model learned from historical admissions, which may reflect unequal past treatment, is a serious concern.
• Transparency. Nurses must see the main reasons for each ranking and warnings when a patient is unlike the training data; patients must be told the model is used.
• Work and responsibility. Nurses must have the authority, time and training to override the model, must not be evaluated by how often they agree with it, and must not become a moral crumple zone. The hospital must name an accountable owner, and the contract with Brightline must allocate liability.
• Alignment. The model predicts past admission to intensive care, which is a proxy for urgency and may encode old patterns. The objective should be changed to measures closer to clinical urgency, and the model retested.
• Governance and law. The hospital should treat the system as a medical device question for Health Canada, follow any provincial rules for public sector AI, such as those Ontario can make under its 2024 law, and provincial health privacy law, and, by analogy with the federal Directive, complete an impact assessment and publish a summary. It should adopt the system in stages, with a sunset date and a clear power to suspend.

The analysis concludes that Pathfinder should not be adopted as proposed, but could be adopted in a limited, monitored trial once four problems are fixed: the training objective, the subgroup error analysis, explanations for nurses and a protected right to override. The board votes to accept that recommendation. Theo, who started the term ready to add up outcomes, says the final recommendation is better than the one he would have made, not because it rejects the numbers but because it asks what they leave out. Sabine says it respects the patients the model gets wrong. Varga says that is what the course has been for.