Organizational AI governance: structures, policies, procurement and contracts

Laws, standards and audits all depend on what happens inside organizations: who decides whether an AI system is built or bought, how risks are assessed, which controls are applied, how staff use AI tools, how vendors are chosen and bound by contract, and what happens when something goes wrong. Organizational AI governance turns external requirements and internal values into everyday practice. This lesson covers governance structures and roles, policies (including acceptable use of generative AI), life cycle review processes, procurement and vendor due diligence, the key clauses of AI contracts, incident response, training and AI literacy, and measuring and reporting on governance.

Structures and roles

Effective AI governance assigns clear responsibilities at several levels:

• Board of directors: oversight of AI strategy and material AI risks, as part of its general duty of oversight. Directors need enough understanding to ask good questions and should receive regular reporting.
• Executive accountability: a senior executive accountable for AI governance, sometimes titled chief AI officer, often working with the chief risk, privacy, legal, information security and technology officers.
• An AI governance committee or council: a cross-functional body (legal, privacy, security, risk, compliance, ethics, engineering, product and business representatives) that sets policy, reviews high-risk systems and resolves escalations.
• System owners: a named business owner accountable for each AI system's purpose, performance and compliance, supported by technical owners.
• The three lines model: the first line (business and development teams) owns and manages risks; the second line (risk, compliance, privacy) sets frameworks and provides oversight and challenge; the third line (internal audit) provides independent assurance.

Smaller organizations can combine roles, but the principle remains: someone specific is accountable for each AI system and for the governance program as a whole.

Policies

An AI governance program usually rests on a small set of documents:

• AI principles: a short statement of commitments (for example fairness, transparency, safety, privacy, accountability) that guides decisions.
• AI policy: the mandatory rules: scope, roles, the inventory and risk-tiering requirements, required assessments and approvals by tier, documentation, monitoring, incident reporting, and exceptions.
• Standards and procedures: detailed requirements for specific activities, such as model validation, bias testing, data use, and third-party AI.
• Acceptable use policy for AI tools: rules for staff using generative AI and other tools, covering approved tools, prohibited uses, protection of confidential and personal information, verification of outputs, disclosure of AI use, and intellectual property.

Policies should be short enough to be read, specific enough to be followed, and aligned with what teams actually do. A policy that bans all use of public generative AI tools, without providing approved alternatives, tends to drive use underground ("shadow AI") rather than stopping it.

Life cycle review

Governance is applied through review points in the AI system life cycle, scaled to risk:

• Intake: a new AI use case is registered in the inventory with a brief description, owner and initial risk tier.
• Design review: for medium and high tiers, impact assessments (privacy, algorithmic, fundamental rights) and a plan for data, testing and oversight.
• Pre-deployment approval: evidence of testing (performance, bias, robustness, security), documentation, human oversight design, and sign-off by the appropriate authority.
• Deployment and monitoring: performance, drift, fairness and incident monitoring with thresholds (Lesson 9).
• Change management: material changes, such as retraining, new data sources, new uses or a new vendor model version, trigger re-review.
• Retirement: decommissioning, data retention and documentation archiving.

Low-tier systems may need only registration and adherence to standard practices; high-tier systems get full review. Integrating these steps into existing product development and procurement workflows, rather than adding a separate bureaucracy, makes them sustainable.

Procurement and vendor due diligence

Most organizations buy more AI than they build, and AI arrives through many channels: dedicated AI products, AI features added to existing software, and foundation model APIs used by internal teams. Third-party AI must be governed as carefully as internal AI, because the deployer usually remains responsible to customers, employees and regulators. Due diligence questions include:

• What does the system do, and what are its intended uses and known limitations?
• What data was it trained on, and will the vendor use our data to train or improve its models?
• What testing has the vendor done for accuracy, bias, robustness and security, and can we see the evidence?
• Where is data stored and processed, and which subprocessors are involved?
• What certifications (ISO/IEC 27001, 42001, SOC 2) and audits does the vendor have?
• How will the vendor notify us of incidents, model changes and new risks?
• Does the vendor comply with applicable AI laws, and what role does it take (for example provider under the EU AI Act)?

Structured scoring helps compare vendors and identify issues to negotiate.

# Scoring AI vendors on governance criteria before procurement (weights sum to 1; scores 0 to 5)
criteria = {"data use and training on customer data": 0.20, "security and certifications": 0.15,
            "documentation and model cards": 0.10, "bias and performance testing evidence": 0.15,
            "incident notification and support": 0.10, "contract terms: liability and indemnity": 0.15,
            "audit rights and transparency": 0.10, "exit and data return": 0.05}
vendors = {
    "Vendor X": [5, 4, 3, 2, 4, 2, 3, 4],
    "Vendor Y": [3, 5, 4, 4, 3, 4, 2, 3],
    "Vendor Z": [2, 3, 2, 1, 2, 5, 1, 2],
}
w = list(criteria.values())
names = list(criteria)
for v, sc in vendors.items():
    total = sum(a * b for a, b in zip(w, sc))
    gaps = [names[i] for i, s in enumerate(sc) if s <= 2]
    gap_text = ", ".join(gaps) if gaps else "none"
    print(f"{v}: weighted score {total:.2f} of 5; gaps to negotiate: {gap_text}")

Vendor X: weighted score 3.40 of 5; gaps to negotiate: bias and performance testing evidence, contract terms: liability and indemnity
Vendor Y: weighted score 3.60 of 5; gaps to negotiate: audit rights and transparency
Vendor Z: weighted score 2.35 of 5; gaps to negotiate: data use and training on customer data, documentation and model cards, bias and performance testing evidence, incident notification and support, audit rights and transparency, exit and data return

Vendor Y scores highest overall, but its audit rights need negotiation; Vendor X is strong on data use but weak on bias evidence and contract terms; Vendor Z, despite generous liability terms, has gaps in almost every governance area and would need substantial improvement to be acceptable for a high-risk use. Scores should inform, not replace, judgment: a single critical gap, such as training on customer data without consent, can disqualify a vendor regardless of its total.

Key clauses in AI contracts

AI contracts build on standard technology agreements, with additional clauses addressing AI-specific risks:

• Data use: whether the vendor may use customer data, prompts and outputs to train or improve models; most enterprise agreements now offer no-training commitments.
• Confidentiality, security and data protection: security controls, data location, retention and deletion, breach notification, and privacy law compliance, including data processing agreements.
• Performance and documentation: descriptions of functionality, service levels, documentation such as model cards, and warranties of conformity with documentation.
• Testing and bias: commitments to test for bias and accuracy, provide results, and cooperate with the customer's audits and impact assessments.
• Change management: notice of material model changes, deprecations and new versions, with the right to test before changes take effect for high-risk uses.
• Incident notification: prompt notice of incidents affecting the customer, including security breaches, harmful outputs and regulatory inquiries.
• Intellectual property: ownership of outputs and customizations, licenses, and indemnities for infringement claims, with their conditions (Lesson 6).
• Compliance and roles: allocation of obligations under applicable AI laws, including cooperation with regulators.
• Liability: caps, exclusions and carve-outs (for example uncapped liability for data breaches or IP indemnities), and insurance requirements (Lesson 7).
• Audit and transparency rights, subcontracting controls, and exit: data return, transition assistance and deletion at termination.

Large vendors' standard terms may be difficult to change, but customers can often negotiate enterprise terms, choose configurations (such as disabling data retention), or add controls on their side where contract terms are weak.

Incident response

AI incidents include harmful or wrong outputs at scale, discriminatory outcomes discovered in monitoring, privacy breaches through models, security attacks such as prompt injection leading to data exfiltration (AI-430), and failures of safety-critical systems. An AI incident response plan, integrated with existing security and privacy incident processes, defines how incidents are detected and reported internally, who triages and decides, how systems are paused or rolled back, how affected people, customers and regulators are notified (with deadlines under privacy and AI laws), how root causes are investigated, and how lessons are fed back into controls. Public incident databases record AI failures across industries and are useful for anticipating risks.

Training and AI literacy

The EU AI Act requires AI literacy for staff dealing with AI systems (Lesson 3), and good governance requires it everywhere. Training should be tailored: all staff need to know the acceptable use policy and the basic limitations of AI tools; developers and data scientists need training on documentation, testing, bias and privacy engineering; procurement and legal staff need training on AI contracts; executives and board members need enough understanding to oversee AI strategy and risk. Training is most effective when practical, using the organization's own tools and examples.

Measuring and reporting

Governance programs need metrics to show whether they are working: the share of AI systems inventoried and risk-tiered, assessments completed on time, systems deployed with required approvals, monitoring alerts and their resolution, incidents and time to resolve, training completion, vendor reviews completed, and audit findings closed. Regular reporting to the governance committee, executives and the board keeps attention on AI risk and supports accountability. Some organizations also publish transparency reports on their AI use and governance.