Automated decisions, discrimination and human rights

When AI systems help decide who is hired, who receives credit, who is approved for housing or insurance, or who is flagged for investigation, they engage some of the most important protections in law: the right to equal treatment without discrimination, and, for governments, the rights to procedural fairness and equality guaranteed by constitutions. Algorithms can discriminate without anyone intending them to, by learning patterns from biased historical data or using proxies for protected characteristics. This lesson explains how algorithmic discrimination arises, the legal concepts of direct and adverse effect discrimination, the frameworks that apply in employment, credit, housing and services, how adverse impact is measured, the obligations of explanation and accommodation, human oversight and its limits, and how organizations can manage these risks.

How algorithms discriminate

Algorithmic discrimination usually arises from data and design choices rather than intent:

• Historical bias: a model trained on past hiring decisions learns the preferences, including biases, of past decision-makers. A widely reported case involved an experimental recruiting tool that learned to downgrade resumes containing words associated with women, because past hires in technical roles were mostly men.
• Proxies: even without protected attributes as inputs, models can use correlated features such as postal code, school, name, employment gaps or purchasing patterns that act as proxies for race, sex, age or disability.
• Representation: groups underrepresented in training data may experience higher error rates, as studies of facial recognition showed for darker-skinned women.
• Measurement: the target variable may itself be biased, such as using arrests as a proxy for crime, or health spending as a proxy for health need (a well-known study found a widely used health algorithm underestimated the needs of Black patients for this reason).
• Feedback loops: decisions influence future data, for example policing directed by predictions generates more data in the same areas.
• Accessibility: AI interviews or tests may disadvantage people with disabilities, such as speech or vision impairments, if accommodations are not provided.

Legal concepts of discrimination

Anti-discrimination law recognizes two broad forms:

• Direct discrimination (disparate treatment): treating someone less favourably because of a protected characteristic, for example a model that uses sex or age as an input to reject applicants, or a rule deliberately targeting a group.
• Adverse effect discrimination (disparate impact): a neutral-seeming rule or practice that disproportionately disadvantages a protected group, which is unlawful unless justified. Most algorithmic discrimination falls into this category.

In Canada, the Canadian Human Rights Act (for federally regulated employers and service providers) and provincial human rights codes prohibit discrimination in employment, services, housing and contracts on grounds such as race, national or ethnic origin, colour, religion, age, sex, sexual orientation, gender identity or expression, marital and family status, and disability. Canadian law does not require intent: a practice with an adverse effect on a protected group is discriminatory unless the respondent shows it is a bona fide requirement, rationally connected to a legitimate purpose, adopted in good faith, and reasonably necessary, with accommodation to the point of undue hardship (the test from the Supreme Court of Canada's Meiorin decision, 1999). For governments, section 15 of the Canadian Charter of Rights and Freedoms guarantees equality, and administrative law requires procedural fairness in decisions affecting individuals, including the right to reasons in many cases. Human rights commissions in Ontario and elsewhere have issued guidance on AI and human rights.

Measuring adverse impact

A common screening measure in employment is the selection rate ratio, used in the four-fifths (80 %) rule of thumb from the US Uniform Guidelines on Employee Selection Procedures (1978): if a group's selection rate is less than 80 % of the highest group's rate, this is generally regarded as evidence of adverse impact. The rule is a guide, not a legal standard in Canada and not conclusive in the United States, where statistical significance and practical significance are both considered. Similar analysis applies to credit approvals, pricing and other decisions.

import numpy as np
# Adverse impact in a hiring screen: selection rates by group and the four-fifths rule
applicants = {"group A": 1200, "group B": 800}
selected = {"group A": 360, "group B": 168}
rates = {g: selected[g] / applicants[g] for g in applicants}
best = max(rates.values())
for g, r in rates.items():
    ratio = r / best
    flag = "below four-fifths" if ratio < 0.8 else "ok"
    print(f"{g}: selection rate {r:.1%}, impact ratio {ratio:.2f} ({flag})")
# Is the difference statistically significant? Two-proportion z test
p1, p2 = rates["group A"], rates["group B"]
n1, n2 = applicants["group A"], applicants["group B"]
p = (selected["group A"] + selected["group B"]) / (n1 + n2)
z = (p1 - p2) / np.sqrt(p * (1 - p) * (1 / n1 + 1 / n2))
print(f"difference {p1 - p2:.1%}, z = {z:.2f}")
# With small samples the same rates may not be significant
z_small = (p1 - p2) / np.sqrt(p * (1 - p) * (1 / 60 + 1 / 40))
print(f"same rates with 60 and 40 applicants: z = {z_small:.2f}")

group A: selection rate 30.0%, impact ratio 1.00 (ok)
group B: selection rate 21.0%, impact ratio 0.70 (below four-fifths)
difference 9.0%, z = 4.47
same rates with 60 and 40 applicants: z = 1.00

Group B is selected at 21 % against 30 % for group A, an impact ratio of 0.70, below four-fifths, and with 2,000 applicants the difference is highly significant (z about 4.5). With only 100 applicants, the same rates would not be statistically significant (z about 1.0), which is why small samples need careful interpretation and why ongoing monitoring across larger volumes is important. Finding adverse impact is not the end of the analysis: the organization must examine why it arises, whether the criteria are job-related and necessary, and whether less discriminatory alternatives exist, such as removing proxy features or adjusting thresholds.

Explanation and reasons

Laws increasingly require explanations of automated decisions. In the United States, the Equal Credit Opportunity Act requires lenders to give applicants specific reasons for adverse credit decisions, which means the model must support accurate identification of the principal factors, not generic statements. Quebec's Law 25, the EU AI Act and GDPR, and several US state laws give rights to information about the main factors in automated decisions. Government decision-makers must generally give reasons that allow people to understand and challenge decisions. Explanations should be accurate (faithful to the model, AI-430), meaningful to the recipient, and actionable where possible, indicating what could change the outcome. Technical explanation methods such as feature attribution help, but must be validated, and complex models may need simpler surrogate or inherently interpretable models for high-stakes decisions.

Accommodation and accessibility

Employers and service providers have a duty to accommodate people with disabilities, and other protected characteristics such as religion, to the point of undue hardship. AI-driven processes, such as automated video interviews, timed online assessments, voice systems and chatbots, can create barriers for people with visual, hearing, speech, motor, cognitive or mental health disabilities. Organizations should inform applicants about AI tools, offer alternative formats or human assessment on request, test tools for accessibility, and ensure that accommodation requests are not screened out by the system. Accessibility laws, such as the Accessible Canada Act and provincial laws, and the ADA in the United States, reinforce these obligations.

Human oversight and its limits

Many frameworks require human oversight of automated decisions, and keeping a human in the loop is often presented as the safeguard against algorithmic harm. But oversight is only meaningful if the human has the authority, time, information and competence to disagree with the system. Research shows automation bias: people tend to accept system recommendations, especially under time pressure, and may defer even when the recommendation is wrong. Rubber-stamp review of hundreds of decisions per hour does not satisfy the purpose of oversight and may not satisfy the law. Effective oversight design includes clear criteria for when to override, information about the system's confidence and limitations, manageable workloads, monitoring of override rates, and training. Legal rules that apply only to decisions made solely by automated means can sometimes be avoided by nominal human involvement, which regulators increasingly scrutinize.

Managing discrimination risk

An organization deploying AI in consequential decisions should:

• Assess impact before deployment: identify protected groups, available data to test outcomes (collecting demographic data for testing where lawful and with safeguards), and potential proxies.
• Test for adverse impact and error rate differences across groups, on representative data.
• Investigate and mitigate: remove or constrain proxy features, rebalance data, adjust thresholds, or choose different models; document the analysis of alternatives.
• Validate job-relatedness or business necessity: show that criteria predict the outcome that matters.
• Provide notice, explanations, accommodation and human review.
• Monitor in production for drift in outcomes across groups, and re-audit regularly, independently where required.
• Require vendors to provide testing evidence and cooperate with audits (Lesson 11).

Fairness metrics and mitigation techniques are covered in depth in AI-460. Technical fairness and legal compliance overlap but are not identical: the right metric depends on the legal context and the decision.

Organizations should also remember that discrimination law protects people throughout the employment relationship, not only in hiring: AI used for scheduling, performance evaluation, promotion, pay and termination decisions raises the same issues and deserves the same testing and oversight.