AI risk management, controls and assurance

Lesson 8 set up the governance structure: principles, roles, an inventory, risk tiers and policy. This lesson is about the work that makes governance real: assessing the risks of a specific AI system, applying controls across its lifecycle, testing for fairness and robustness, monitoring it in use, handling AI incidents, managing AI suppliers and giving assurance to the board and regulators. It follows Larchmont's two hardest cases: the payments fraud score supplied by a vendor and the paused résumé screening feature.

AI risks

AI risks can be grouped in several ways. A practical grouping for service managers covers harms to people (unfair treatment, privacy intrusion, safety, loss of autonomy), harms to the organization (wrong decisions, financial loss, legal liability, reputational damage, operational disruption) and harms to society (misinformation, environmental cost, concentration of power). Many risks cross groups: a biased screening tool harms applicants, exposes the employer to legal claims and damages trust.

The AI impact assessment

An AI impact assessment (sometimes called an algorithmic impact assessment) examines a proposed AI system before deployment: its purpose and necessity, the people it affects, the data it uses, how it could cause harm, how likely and severe those harms are, and what controls will reduce them. It is usually combined with or linked to a privacy impact assessment. Canada's federal Directive on Automated Decision-Making requires an Algorithmic Impact Assessment questionnaire for federal systems, and ISO/IEC 42005 offers international guidance; many private organizations adapt these.

Larchmont's assessment template has eight parts: purpose and alternatives (could the goal be met without AI?), people affected and how, data sources and lawfulness, performance requirements and testing plan, fairness analysis, transparency and explanation, human oversight design, and monitoring and exit plan. High-risk systems must complete all eight with sign-off from the system owner, privacy and the AI governance group; medium-risk systems complete a shorter version.

Lifecycle controls

AI controls work best when spread across the lifecycle. Before building or buying, the impact assessment and approval decide whether to proceed. Data controls cover lineage (where data came from), quality, representativeness, lawful basis and protection. Development or procurement controls require documentation, often in the form of model cards or system cards that describe intended use, training data, performance, limitations and known risks. Validation tests accuracy, fairness, robustness and misuse before go-live. Deployment controls include notices to people affected, human oversight design and the ability to roll back. Operating controls cover monitoring, drift detection, incident handling and periodic review. Retirement controls handle data and dependencies when the system is withdrawn.

These map naturally onto ITIL practices. Change enablement controls model updates and new versions; release and deployment management handle progressive rollout; monitoring and event management watch performance and drift; incident and problem management handle failures; supplier management covers vendors; information security management covers threats; and continual improvement uses findings to improve systems. AI governance does not need a separate operational world.

Testing for fairness

Fairness testing compares how a system performs or what outcomes it produces for different groups of people. Common checks include comparing selection or approval rates across groups, comparing error rates (for example, how often legitimate payments are wrongly blocked for each group) and examining which input features drive outcomes. Some features, such as postal code or school attended, can act as proxies for protected characteristics like race or ethnicity even when those characteristics are not used directly.

Fairness is not a single number. Different fairness measures can conflict: equal selection rates, equal error rates and equal accuracy cannot always all be achieved at once when groups differ in their underlying data. Choosing which measures matter for a given use is a judgment that involves the business, legal advice and the people affected, and it should be documented.

# Selection-rate ratios and a simple drift check (invented data)
groups = {"urban": (180, 600), "rural": (64, 400), "applied in French": (22, 90)}
rates = {g: sel / n for g, (sel, n) in groups.items()}
best = max(rates.values())
for g, r in rates.items():
    print(f"{g:<18} selection {r * 100:5.1f} %   ratio to highest {r / best:.2f}")

# Population stability index (PSI) for a fraud score distribution: training vs this month
import math
bins_train = [0.40, 0.30, 0.15, 0.10, 0.05]
bins_now   = [0.30, 0.28, 0.18, 0.14, 0.10]
psi = sum((n - t) * math.log(n / t) for t, n in zip(bins_train, bins_now))
print(f"PSI = {psi:.3f}  (rules of thumb: <0.1 stable, 0.1-0.25 watch, >0.25 investigate)")

urban              selection  30.0 %   ratio to highest 1.00
rural              selection  16.0 %   ratio to highest 0.53
applied in French  selection  24.4 %   ratio to highest 0.81
PSI = 0.084  (rules of thumb: <0.1 stable, 0.1-0.25 watch, >0.25 investigate)

In the United States, Title VII of the Civil Rights Act, the Age Discrimination in Employment Act and the ADA prohibit employment discrimination, including policies with a disparate impact that cannot be justified. The long-standing 'four-fifths rule' from the 1978 Uniform Guidelines treats a selection rate for any group below 80 % of the highest group's rate as evidence of adverse impact, a rule of thumb rather than a legal test. A 2025 executive order directed federal agencies to move away from disparate-impact enforcement, but private lawsuits and state laws still apply, and laws such as New York City's Local Law 144 require bias audits and notices for automated employment decision tools.

Explainability and transparency

Explainability means being able to give meaningful reasons for an AI-influenced outcome at the level the audience needs. A customer whose payment was blocked needs a plain reason and a way to resolve it; an analyst needs to know which signals drove a fraud score; an auditor needs to know how the model was validated. Some models are inherently interpretable, such as simple scorecards; others need explanation techniques that estimate which inputs mattered. Explanations should be tested with real users to check that they help.

Transparency also means telling people when AI is involved. Larchmont's policy requires notices when customers interact with the virtual agent, when AI materially influences a decision about a customer or applicant, and when content is AI-generated in ways a reasonable person would want to know. It also requires an accessible route to a human review of significant decisions.

Monitoring AI in operation

Once deployed, AI systems need monitoring on several fronts: performance (accuracy, precision and recall, measured against ground truth as it becomes available), fairness (outcomes by group over time), data drift (whether inputs still look like the training data, measured with statistics such as the population stability index), usage (whether people use and override it as expected) and incidents and complaints. Each monitored measure needs a threshold and an owner who acts when it is crossed.

The payments fraud score illustrated why. Three months after Larchmont launched a new corporate card payment option, the share of payments blocked for review doubled and the population stability index rose to about 0.24. The model had never seen corporate card patterns. Because monitoring caught it, Larchmont and the vendor adjusted thresholds within a week and scheduled retraining, rather than discovering the problem through angry customers.

AI incidents

An AI incident is an event in which an AI system causes or nearly causes harm: a virtual agent gives dangerous advice, a model makes systematically unfair decisions, an AI feature leaks data, or an automated action disrupts a service. AI incidents should flow through the normal incident management practice with additional steps: preserve inputs and outputs for investigation, consider switching the AI off or reverting to a manual process, assess whether people were harmed and need to be informed, and check whether legal or regulatory reporting applies, for example privacy breach reporting.

Problem management then looks for causes, which may lie in data, model design, prompts, integration, oversight or process. Public databases such as the AI Incident Database, run by the Responsible AI Collaborative, collect reported incidents from many organizations and are useful for learning what has gone wrong elsewhere.

Managing AI suppliers

Most of Larchmont's AI comes from suppliers, so supplier management carries much of the risk. Due diligence asks how the AI was built and tested, what data it was trained on, how it performs on data like Larchmont's, how bias was assessed, how customer data is used and protected, whether the supplier holds relevant certifications such as ISO/IEC 42001 or ISO/IEC 27001, and how incidents and model changes are handled.

Assurance

The board needs confidence that AI governance works. Assurance comes in layers, often described as three lines: system owners and teams manage risk day to day; risk, compliance, privacy and security functions set standards and challenge; internal audit gives independent assurance. External assurance can add certification to ISO/IEC 42001 or independent audits of specific systems, which some laws require.

Larchmont's AI governance group reports to the board's risk committee each quarter with the inventory by risk tier, new approvals, assessments completed, monitoring alerts and incidents, and progress on open actions. Internal audit included AI governance in its annual plan for the first time, testing a sample of three systems from registration to monitoring.