Reliability, security and governance

As ML systems take on more important decisions, such as approving loans, flagging fraud, prioritizing patients or screening job applicants, the bar rises. They must be reliable enough that the business can depend on them, secure against people who want to manipulate or steal them, fair to the people they affect, and governed so that the organization knows what models it runs, what risks they carry and who is accountable. This lesson covers reliability engineering for ML services, the security threats specific to ML, privacy and fairness in operation, and the frameworks and regulations for model governance in Canada and the United States.

Service level objectives and error budgets

Site reliability engineering (SRE), developed at Google, gives a disciplined way to balance reliability against the pace of change. A service level indicator (SLI) is a measured quantity, such as the share of requests answered successfully within 100 ms. A service level objective (SLO) is a target for it, such as 99.9 % over 30 days. The error budget is what the SLO allows to go wrong: 0.1 % of the time, about 43 minutes in 30 days. While budget remains, teams can release changes freely; when it is exhausted, releases pause and effort shifts to reliability.

import numpy as np

# 1. Error budget for a model endpoint with a 99.9 % availability SLO over 30 days
slo, minutes = 0.999, 30 * 24 * 60
budget = (1 - slo) * minutes
incidents = [("bad deploy, rolled back", 18), ("feature store outage", 9), ("GPU node failure", 4)]
used = sum(m for _, m in incidents)
print(f"Error budget: {budget:.1f} min; used {used} min ({used / budget:.0%}); remaining {budget - used:.1f} min")

# 2. Fairness monitoring: approval rates and error rates by group on last month of decisions
rng = np.random.default_rng(11)
groups = {"A": (6000, 0.30), "B": (2500, 0.38)}       # (applicants, share who would default)
for g, (n, bad) in groups.items():
    will_default = rng.random(n) < bad
    score = rng.normal(np.where(will_default, 0.40, 0.65), 0.15)
    if g == "B":
        score -= 0.05                                    # a feature acts as a proxy for group
    approve = score > 0.52
    tpr = approve[~will_default].mean()                  # good applicants approved
    print(f"Group {g}: approval rate {approve.mean():.1%}, good applicants approved {tpr:.1%}, "
          f"defaulters approved {approve[will_default].mean():.1%}")

Error budget: 43.2 min; used 31 min (72%); remaining 12.2 min
Group A: approval rate 62.4%, good applicants approved 80.5%, defaulters approved 21.0%
Group B: approval rate 48.8%, good applicants approved 69.6%, defaulters approved 13.1%

With three incidents totalling 31 minutes, the endpoint has used 72 % of its monthly budget, leaving about 12 minutes. A sensible policy would slow risky releases for the rest of the month, and the incident list shows where to invest: deployments caused most of the downtime, so better canary analysis (Lesson 5) would pay off. For ML services, SLOs should cover not only availability and latency but also model quality, for example "the daily share of predictions computed with complete features is above 99.5 %" or "weekly precision on labelled samples above 0.8", because a service that answers quickly with wrong predictions is not reliable.

Incidents in ML systems

ML incidents have recognizable causes: bad deployments, upstream data changes, feature store or pipeline outages, drift, and feedback loops. The response follows the general incident pattern: detect, mitigate, communicate, resolve, learn. The fastest mitigations are usually to roll back to the previous model version (Lesson 5), switch to a fallback such as a rule-based policy, or pause automated decisions. After the incident, a blameless post-incident review identifies contributing causes and system improvements, such as a missing validation rule or alert, rather than individual blame. Runbooks for each model (Lesson 7) make the first minutes of an incident faster.

Security threats to ML systems

ML systems inherit every ordinary software threat and add some of their own. Frameworks such as MITRE ATLAS and the OWASP lists for machine learning and LLM applications catalogue them. The main categories are:

• Data poisoning: an attacker who can influence training data, for example through user-generated content, feedback buttons or a compromised data source, inserts examples that degrade the model or plant a backdoor triggered by a specific pattern.
• Evasion: crafting inputs at inference time that the model misclassifies, such as fraud transactions shaped to look normal, or adversarial perturbations to images.
• Model extraction and inversion: querying a model many times to copy its behaviour, or to infer sensitive information about its training data (membership inference).
• Supply chain: malicious or vulnerable dependencies, pretrained models and datasets from untrusted sources, including pickled models that execute code when loaded (Lesson 5).
• Prompt injection and tool misuse: in applications built on large language models, instructions hidden in user input or retrieved documents can redirect the model's behaviour, a threat discussed in AI-410, Lesson 11.

Defences combine ordinary security (access control, secrets management, encryption, dependency scanning, signed artifacts, network isolation of training and serving) with ML-specific measures: validating and monitoring training data provenance, limiting and monitoring query rates, returning only the information needed (a decision rather than full probabilities), robustness testing, and red-teaming high-risk models.

Privacy in operation

Privacy obligations continue after training. Requests to delete personal data must reach training sets, feature stores, logs and, in some interpretations, models trained on the data, which argues for retention limits and regular retraining from current data. Prediction logs often contain personal data and need the same protection as the source systems. Techniques such as pseudonymization, aggregation, differential privacy (adding calibrated noise so that no individual's data noticeably affects results) and federated learning (training on data that stays on devices) reduce exposure. The privacy laws described in Lesson 2 apply throughout.

Fairness in operation

A model that was fair on its test set can become unfair in production as populations and data change. Fairness should therefore be monitored like accuracy. Common measures compare groups on selection rates (demographic parity), on error rates such as the true positive rate among those who deserve a positive outcome (equal opportunity), or on both error rates (equalized odds). These measures can conflict, and which matters depends on the decision and the law.

In the output, group B is approved 48.8 % of the time against 62.4 % for group A, a ratio of 0.78. Part of that gap reflects a higher default rate in group B, but the error rates show more: among applicants who would repay, 80.5 % of group A are approved but only 69.6 % of group B. The model is harder on creditworthy members of group B, because a feature acts as a proxy for group membership. Investigating the proxy, testing invariance (Lesson 4), and reviewing the decision threshold by group with legal advice are the next steps. A common screening rule in US employment practice treats a selection rate ratio below four-fifths (0.8) as evidence of adverse impact that needs justification; in lending and other domains, other legal standards apply.

Governance

Model governance is the organizational system that keeps track of models and their risks. Its typical elements are:

• A model inventory: every model in use, its purpose, owner, risk tier, data, version and status, linked to the registry (Lesson 3).
• Risk tiering: classifying models by the impact of their errors, so that a model setting credit limits gets far more scrutiny than one ordering marketing emails.
• Independent validation: for higher-risk models, review by people who did not build the model, covering conceptual soundness, data, performance, limitations and monitoring.
• Documentation: model cards and validation reports (Lesson 3), decision logs and change records.
• Ongoing monitoring and periodic review, with clear criteria for retraining, recalibration or retirement.
• Human oversight and contestability: for decisions that significantly affect people, the ability to explain decisions, review them and correct errors.

In Canada, federally regulated financial institutions follow OSFI Guideline E-23, Model Risk Management, whose revised version, explicitly covering AI and machine learning models, takes effect in 2027. The federal government's Directive on Automated Decision-Making requires algorithmic impact assessments for government systems. The proposed Artificial Intelligence and Data Act died when Parliament was prorogued in January 2025, so private-sector AI is governed mainly through privacy law (PIPEDA, Quebec's Law 25), human rights law and sector regulators, with voluntary codes of conduct for generative AI.

Organizations operating internationally also consider the EU AI Act, which classifies AI systems by risk and imposes requirements on high-risk systems, and ISO/IEC 42001, a management system standard for AI. Whatever the regime, the evidence regulators ask for is the evidence good MLOps produces anyway: lineage, testing, monitoring, documentation and controlled change.

Explainability

People affected by significant automated decisions often have a right, or at least a reasonable expectation, to know why. Lenders must give reasons for adverse credit decisions, and privacy laws increasingly require transparency about automated decision-making. Techniques for explaining individual predictions include inherently interpretable models (scorecards, small trees, generalized additive models), feature attribution methods such as SHAP values, and counterfactual explanations ("the application would have been approved if the debt ratio were below 0.4"). Explanations must be faithful to the model and understandable to the person receiving them, and they should be tested like any other output, because a misleading explanation can be worse than none.

Explanations are also valuable internally: validators use them to check that a model relies on sensible factors, and operations teams use them to understand surprising predictions during incidents. Recording the explanation alongside each high-impact decision makes later review and appeals much easier.