Defining and measuring fairness

To reduce unfairness we first need to measure it, and measurement requires precise definitions. Researchers have proposed dozens of formal fairness criteria. Most fall into a few families: group fairness criteria that compare statistics across groups, individual fairness criteria that require similar people to be treated similarly, and causal criteria that ask whether a sensitive attribute influenced a decision. Each captures a different intuition about what fairness means, and, as Lesson 4 shows, they cannot all be satisfied at once. This lesson defines the main criteria, shows how to compute them, and discusses when each is appropriate.

Notation

We consider a binary classifier with prediction \(\hat{Y}\) (1 means the favourable or flagged outcome, depending on context), true outcome \(Y\), risk score \(S\), and sensitive attribute \(A\) taking values such as \(a\) and \(b\). Most group fairness criteria can be written as conditional independence statements involving these variables, and they are usually reported as differences or ratios between groups.

• Selection rate: \(P(\hat{Y}=1 \mid A=a)\), the share of the group receiving the positive prediction.
• True positive rate (recall, sensitivity): \(P(\hat{Y}=1 \mid Y=1, A=a)\).
• False positive rate: \(P(\hat{Y}=1 \mid Y=0, A=a)\).
• Positive predictive value (precision): \(P(Y=1 \mid \hat{Y}=1, A=a)\).
• Base rate: \(P(Y=1 \mid A=a)\), the actual prevalence of the outcome in the group.

Demographic parity

Demographic parity, also called statistical parity or independence, requires equal selection rates across groups: \(\hat{Y}\) is independent of \(A\).

It is often reported as a difference (the demographic parity difference) or a ratio (the disparate impact ratio). The four-fifths rule from US employment practice flags a ratio below 0.8. Demographic parity ignores the true outcome, which is both its strength and its weakness. It is appropriate when the labels themselves are believed to be biased, or when the goal is equal access to an opportunity regardless of measured qualifications. It can be inappropriate when base rates genuinely differ and the decision should track them, for example in medical diagnosis, where forcing equal positive rates would mean missing disease in one group or over-treating another.

Equalized odds and equal opportunity

Equalized odds, or separation, requires equal true positive rates and equal false positive rates across groups: \(\hat{Y}\) is independent of \(A\) given \(Y\).

In words, among people who truly have the outcome, each group is equally likely to be correctly identified; among people who do not, each group is equally likely to be wrongly flagged. Equal opportunity is the relaxation that requires only equal true positive rates, appropriate when the positive prediction is a benefit and the main concern is that qualified people in every group get it. Equalized odds was the criterion at the centre of the debate over the COMPAS recidivism tool: a ProPublica analysis found that Black defendants who did not reoffend were almost twice as likely as white defendants to be labelled high risk, a false positive rate disparity.

Predictive parity and calibration

Predictive parity requires equal positive predictive value across groups: among people flagged, the same share in each group truly have the outcome. Calibration within groups, or sufficiency, is the stronger requirement that for every score value \(s\), the probability of the outcome is the same in each group:

Calibration means a score of 0.7 means the same thing for everyone: 70 % of people with that score have the outcome, whatever their group. Decision-makers often care about calibration because they act on the score as a probability. The developer of COMPAS argued that its scores were calibrated across races, which was true, and that a high-risk label meant roughly the same reoffence probability for Black and white defendants. Both sides were right about their chosen metric, which is exactly the conflict explored in Lesson 4.

Computing the metrics

All these group metrics come from the confusion matrix computed separately for each group. The following code computes them for a simulated classifier where the two groups have different base rates.

import numpy as np
# Computing common group fairness metrics from predictions
rng = np.random.default_rng(3)
n = 20_000
group = rng.choice(["A", "B"], n, p=[0.6, 0.4])
y = rng.random(n) < np.where(group == "A", 0.30, 0.20)               # base rates differ
score = np.clip(0.45 * y + rng.normal(0.28, 0.15, n), 0, 1)
pred = score > 0.5
rows = {}
for g in ("A", "B"):
    m = group == g
    yy, pp = y[m], pred[m]
    rows[g] = dict(sel=pp.mean(), tpr=pp[yy].mean(), fpr=pp[~yy].mean(), ppv=yy[pp].mean(), base=yy.mean())
print("metric                    A      B      gap")
names = [("base rate", "base"), ("selection rate", "sel"), ("true positive rate", "tpr"),
         ("false positive rate", "fpr"), ("precision (PPV)", "ppv")]
for label, k in names:
    a, b = rows["A"][k], rows["B"][k]
    print(f"{label:24s}  {a:.3f}  {b:.3f}  {a - b:+.3f}")
dp_ratio = rows["B"]["sel"] / rows["A"]["sel"]
print(f"demographic parity ratio  {dp_ratio:.2f}")

metric                    A      B      gap
base rate                 0.296  0.202  +0.094
selection rate            0.330  0.245  +0.085
true positive rate        0.938  0.933  +0.006
false positive rate       0.074  0.070  +0.004
precision (PPV)           0.843  0.771  +0.071
demographic parity ratio  0.74

The classifier treats people the same way given their true outcome: true positive rates are 0.938 and 0.933 and false positive rates 0.074 and 0.070, so equalized odds is very nearly satisfied. Yet the selection rate of group B is only 74 % of group A's, failing the four-fifths rule, and precision is lower for group B (0.771 against 0.843). The selection gap arises because group B's base rate is lower (0.20 against 0.30), and the precision gap arises because the same false positive rate applied to a group with fewer true positives produces a larger share of false alarms among those flagged. One classifier, three different verdicts, depending on which criterion is chosen.

Individual fairness

Group criteria can be satisfied while individuals are treated arbitrarily: a classifier could achieve demographic parity by accepting the most qualified applicants from one group and random applicants from another. Individual fairness, proposed by Dwork and colleagues, requires that similar individuals receive similar predictions: if a distance \(d(x, x')\) between two people is small, the difference in their outcomes must be small too. The difficulty is defining the similarity metric, which encodes exactly the value judgements about which differences are relevant. Individual fairness is nevertheless a useful lens, and approximate versions are tested by comparing predictions for pairs of nearly identical cases.

Counterfactual and causal fairness

Causal approaches ask whether the sensitive attribute caused the decision. Counterfactual fairness, defined by Kusner and colleagues, requires that a person's prediction would be the same in a counterfactual world where their sensitive attribute was different but everything not caused by it stayed the same. This needs a causal model of how the attribute influences other variables, which is often contested. Causal reasoning helps distinguish paths considered legitimate (an attribute affecting an outcome through a relevant qualification) from those considered discriminatory (through a proxy such as postal code), and it underlies practical tests such as swapping names in résumés or prompts (Lesson 6).

Intersectionality and small groups

People belong to several groups at once, and harms can concentrate at intersections, as in the Gender Shades finding that errors were highest for darker-skinned women, more than either darker-skinned people or women overall. Fairness audits should therefore examine intersections such as sex by race or age by disability, not only one attribute at a time. Intersections create small groups, and metrics estimated from a few dozen people are noisy. Reporting confidence intervals, for example via bootstrap resampling, and treating differences that fall within the noise with caution, avoid both false alarms and false reassurance.

Choosing a criterion

Choosing among criteria is a normative decision informed by the context. Useful questions include: Is the positive prediction a benefit (a loan) or a burden (a fraud investigation)? Which errors cause the most harm, and to whom? Are the labels trustworthy, or do they reflect biased past decisions? Will decision-makers interpret scores as probabilities? Does the law in this domain prescribe a test? Writing the rationale down, consulting affected stakeholders and reporting several metrics, not only the chosen one, makes the choice transparent and contestable.

In Canada, human rights tribunals and courts assess adverse effect discrimination, where a neutral rule has a disproportionate impact on a protected group, and the employer or service provider must then justify it; no fixed numeric threshold like the four-fifths rule applies, so organizations typically report several metrics and document their justification.

Estimating metrics reliably

Fairness metrics are statistics estimated from samples, so they carry uncertainty. A true positive rate computed from 40 positive examples in a small group has a standard error of about 0.06 when the rate is near 0.8, so an observed gap of 0.05 between groups may be noise. Bootstrap resampling gives a simple, general way to quantify this: resample the evaluation set with replacement many times, recompute each metric and each gap, and report the 2.5th and 97.5th percentiles as a 95 % interval. When several groups and metrics are compared at once, some differences will look significant by chance, so audits should either correct for multiple comparisons or treat isolated small differences as prompts for further investigation rather than conclusions.

Sample size planning helps too. Before an audit, estimate how many examples from each group are needed to detect a gap of a size that matters, for example 0.05 in true positive rate, and collect or oversample accordingly. An audit that cannot detect meaningful disparities because its groups are too small gives false reassurance, which is worse than acknowledging that the question cannot yet be answered.

Metrics for scores and rankings

Many systems output scores or rankings rather than binary decisions. For scores, fairness can be assessed with group-wise ROC curves and areas under the curve, which show whether the score separates positives from negatives equally well in each group, and with calibration curves plotted per group. For rankings, such as search results or candidate shortlists, exposure-based metrics compare the attention each group receives in the top positions, since items at the top of a list receive far more clicks than those lower down. A ranking can contain both groups in equal numbers yet place one group consistently lower, which an exposure metric reveals and a simple count does not.

For regression tasks, such as predicting a salary offer or a credit limit, the analogues compare mean predictions, mean errors and error variances across groups. A model that systematically under-predicts for one group, even with the same average absolute error, has a fairness problem that the overall error hides.