Safety frameworks, policy and governance

Technical safety methods matter only if organizations use them consistently and decide well about when to train, deploy, restrict or pause models. Governance is the set of commitments, processes, laws and institutions that shape those decisions. For frontier AI it operates at several levels: internal safety frameworks adopted by developers, voluntary commitments and industry codes, national laws and regulators, government safety institutes, and international cooperation. This lesson surveys each level, explains how capability thresholds link evaluations (Lessons 8 and 9) to required safeguards, compares the main legal regimes that apply to frontier developers, and discusses the open questions that make AI governance a fast-moving field. Laws and institutions in this area change frequently; the details below reflect the situation in late 2026, and practitioners should always check current sources.

Frontier safety frameworks

Most leading developers have published frontier safety frameworks: documents that describe which dangerous capabilities they track, how they evaluate them, and which safeguards they commit to apply as capabilities grow. Examples include Anthropic's Responsible Scaling Policy, OpenAI's Preparedness Framework and Google DeepMind's Frontier Safety Framework, and at the 2024 Seoul AI summit, sixteen companies committed to publish frameworks of this kind. Despite differences in terminology, the frameworks share a common structure:

• Risk domains: typically chemical, biological, radiological and nuclear (CBRN) weapons, offensive cyber operations, autonomous AI research and development, and sometimes persuasion or loss of control.
• Capability thresholds: defined levels of dangerous capability in each domain, such as the ability to provide meaningful uplift to a novice attempting to produce a biological weapon, or to automate the work of an entry-level AI researcher.
• Evaluations: tests run before and during training and before deployment to determine whether a model is near or above each threshold, with safety margins to account for elicitation gaps.
• Required safeguards: deployment protections (classifiers, monitoring, access restrictions) and security protections (protecting model weights against theft) that must be in place before a model above a threshold is deployed or further trained.
• Governance: who makes decisions, how they are documented, how the framework is updated, and what external review or reporting takes place.

The core idea is if-then commitments: if evaluations show a capability above a threshold, then specific safeguards are required, and if those safeguards cannot be implemented, deployment or training is paused. Frameworks have been criticized for vague thresholds, discretion in interpretation and the absence of external enforcement, and several have been revised as experience accumulated; newer laws now require some of their elements to be published and followed.

Thresholds based on compute

Laws often need a simple, objective trigger for which models they cover. Training compute, measured in floating-point operations (FLOPs), is widely used because it correlates with capability, is known to the developer before deployment, and is hard to game without also limiting capability. For dense transformers, training compute is approximately six times the number of parameters times the number of training tokens:

# Training compute thresholds used in AI rules. Compute is estimated as about 6 x parameters x tokens.
models = [
    ("Small open model", 8e9, 15e12),
    ("Mid-size model", 70e9, 15e12),
    ("Large model", 400e9, 30e12),
    ("Frontier run", 2e12, 40e12),
]
thresholds = [("EU AI Act systemic-risk presumption", 1e25), ("California SB 53 and New York RAISE", 1e26)]
for name, params, tokens in models:
    flops = 6 * params * tokens
    hits = [label for label, t in thresholds if flops > t]
    print(f"{name:17s} {params:8.1e} parameters {tokens:8.1e} tokens  compute {flops:8.1e}")
    print("    above:", ", ".join(hits) if hits else "no threshold")

Small open model   8.0e+09 parameters  1.5e+13 tokens  compute  7.2e+23
    above: no threshold
Mid-size model     7.0e+10 parameters  1.5e+13 tokens  compute  6.3e+24
    above: no threshold
Large model        4.0e+11 parameters  3.0e+13 tokens  compute  7.2e+25
    above: EU AI Act systemic-risk presumption
Frontier run       2.0e+12 parameters  4.0e+13 tokens  compute  4.8e+26
    above: EU AI Act systemic-risk presumption, California SB 53 and New York RAISE

The small and mid-size models fall below all thresholds. A 400-billion-parameter model trained on 30 trillion tokens uses about 7 × 10^25 FLOPs, above the EU's presumption threshold for systemic risk but below the 10^26 threshold used in the California and New York laws. Only the largest run crosses both. Compute thresholds are crude: algorithmic progress means that a fixed threshold captures increasingly capable models less well over time, fine-tuning and inference-time compute are not always counted, and small specialized models can be dangerous in narrow domains. For these reasons, compute thresholds are usually combined with capability evaluations and with criteria based on the developer's size or revenue.

Laws and regulators

The European Union's AI Act, which entered into force in August 2024, is the most comprehensive horizontal AI law. It regulates high-risk applications (such as AI in hiring, credit and critical infrastructure) and imposes obligations on providers of general-purpose AI (GPAI) models that have applied since August 2025: technical documentation, information for downstream providers, a copyright policy and a summary of training content. Models with systemic risk, presumed above 10^25 FLOPs of training compute or designated by the Commission, face additional duties: model evaluation including adversarial testing, assessment and mitigation of systemic risks, serious incident reporting and adequate cybersecurity. A General-Purpose AI Code of Practice, published in 2025 and signed by most major developers, describes how these obligations can be met; the AI Office oversees compliance.

In Canada, there is no federal law specifically regulating frontier AI. The proposed Artificial Intelligence and Data Act, part of Bill C-27, died when Parliament was prorogued in January 2025. Governance relies instead on existing law (privacy legislation such as PIPEDA and Quebec's Law 25, human rights and competition law), the federal Voluntary Code of Conduct on the Responsible Development and Management of Advanced Generative AI Systems, which many companies have signed, and the Directive on Automated Decision-Making for government systems. Developers serving Canadians who also operate in the United States or the European Union must comply with those jurisdictions' frontier rules, which in practice shape the safety processes applied to models used in Canada.

Across these regimes, recurring elements are transparency (publishing frameworks, model cards and system cards), testing before deployment, incident reporting, cybersecurity for model weights, and whistleblower protections, rather than detailed technical rules, which would quickly become outdated.

Safety institutes and international cooperation

Several governments have created AI safety institutes, technical bodies that evaluate frontier models, conduct research and advise policymakers. The United Kingdom created the first in 2023 (renamed the AI Security Institute in 2025). The US institute, created at NIST in 2023, was reorganized in 2025 as the Center for AI Standards and Innovation (CAISI), focused on standards, voluntary testing agreements with developers and national security risks. Canada launched the Canadian AI Safety Institute (also abbreviated CAISI) in November 2024 within Innovation, Science and Economic Development Canada, funding research through partners including the CIFAR and the National Research Council. These bodies form an international network that shares evaluation methods and, in some cases, jointly tests models before release.

International cooperation also runs through summits (Bletchley Park 2023, Seoul 2024, Paris 2025 and India 2026), the OECD AI Principles and its reporting framework for the Hiroshima AI Process code of conduct, the Council of Europe's Framework Convention on AI (the first binding international treaty on AI, opened for signature in 2024 and signed by Canada, the United States and others), and the International AI Safety Report, a scientific assessment of general-purpose AI risks led by Yoshua Bengio with experts nominated by over thirty countries, first published in January 2025.

Inside organizations

Effective governance inside a developer depends on more than a written framework. Good practice includes a clear decision-making process for training and deployment with documented rationales; independent internal review, for example by a safety or risk committee with authority to delay launches; board-level oversight of catastrophic risk; external evaluation by safety institutes and third-party assessors before major releases; secure channels for employees to raise concerns, protected by law in some jurisdictions; publication of system cards that report evaluation results, safeguards and known limitations; and post-deployment monitoring with incident response (Lesson 10). Security of model weights deserves particular attention, because a stolen model escapes every deployment safeguard; frameworks increasingly specify security levels matched to capability, up to protection against well-resourced state actors.

Open questions

AI governance is unsettled. Key debates include:

• Open-weight models: releasing weights enables research, competition and local control, but safeguards can be removed by fine-tuning and releases cannot be recalled. Many frameworks set capability thresholds above which open release would not be appropriate.
• Federal versus state rules: in the United States, whether a national standard should preempt state frontier AI laws remains contested.
• Verification: how outsiders can confirm that developers follow their own frameworks, and whether audits and third-party evaluation should be mandatory.
• Competition and coordination: unilateral safety commitments may be costly if competitors do not make them, which motivates industry codes, legal minimums and international agreements.
• Compute governance: tracking advanced chips and large training runs as a basis for oversight, including export controls on advanced semiconductors.
• Pace: rules written for today's systems may not fit future ones, which favours flexible, process-based requirements and regular review.

Reading a frontier safety framework critically

When evaluating a developer's published framework, useful questions include: Are the capability thresholds specific enough that an outsider could tell whether they had been crossed? Are the required safeguards concrete, with standards for testing them? Who decides, and can the decision be overridden for commercial reasons? Is there external review or reporting of results? What happens if the safeguards cannot be achieved? Frameworks that answer these questions clearly provide much stronger assurance than those that describe intentions in general terms.