Fairness in language and generative models

The fairness metrics of Lessons 3 to 5 were designed for classifiers that make a decision about a person. Large language models and image generators are different: they produce open-ended text and images, serve many purposes at once, and are often adapted by others for tasks their developers never anticipated. Their harms are frequently representational, such as stereotypes, demeaning content or the erasure of groups, but they also cause allocative harms when used for screening résumés, grading essays or triaging support requests, and quality-of-service harms when they work less well in some languages or dialects. This lesson covers how bias shows up in generative models, how to measure it, and how developers and deployers mitigate it.

How bias appears

Generative models learn statistical patterns from enormous corpora of web text, books, code and images. Those corpora over-represent some languages, regions and viewpoints, and they contain stereotypes, slurs and historical prejudices. Common manifestations include:

• Stereotyped associations: completing "the nurse said that" with "she" and "the engineer said that" with "he", or depicting certain nationalities in stereotyped settings.
• Unequal sentiment and toxicity: generating more negative or toxic continuations for prompts mentioning some identities.
• Default assumptions: producing images of light-skinned men for neutral prompts like "a CEO" or "a doctor", erasing others.
• Dialect and language disparities: performing worse in low-resource languages, or rating text written in African American English as less professional or more toxic.
• Allocative effects in downstream use: ranking résumés differently depending on the name at the top, a pattern found in several audits of LLM-based screening.
• Over-correction: refusing reasonable requests mentioning some groups, or producing historically inaccurate images in an attempt to diversify, which is also a failure.

Counterfactual testing

The most widely used measurement technique is counterfactual testing, also called perturbation testing: create templates or real prompts, vary only the identity term (a name, a pronoun, a nationality, a religion), and compare the outputs. If the system is fair in this sense, the outputs should not change in ways that matter. For classifiers built on language models, compare scores; for open-ended generation, compare sentiment, toxicity, refusal rates, or task outcomes such as which candidate is recommended.

import numpy as np
# Counterfactual template testing of a toy text scorer that learned biased word weights
weights = {"excellent": 2.0, "reliable": 1.2, "late": -1.0, "rude": -1.8, "helpful": 1.5,
           "engineer": 0.3, "nurse": 0.3, "he": 0.25, "she": -0.10, "they": 0.0,
           "emma": -0.15, "james": 0.20, "aisha": -0.35, "wei": -0.20, "carlos": -0.25}
def score(text):
    toks = text.lower().replace(".", "").split()
    s = sum(weights.get(t, 0.0) for t in toks)
    return 1 / (1 + np.exp(-s))
templates = ["NAME is a reliable engineer.", "NAME was late but helpful.", "NAME is an excellent nurse.",
             "PRON was rude to the client.", "PRON is a helpful engineer."]
names = ["James", "Emma", "Aisha", "Wei", "Carlos"]
prons = ["He", "She", "They"]
print("template                          " + "  ".join(f"{v:>6s}" for v in names))
gaps = []
for t in templates[:3]:
    vals = [score(t.replace("NAME", v)) for v in names]
    gaps.append(max(vals) - min(vals))
    print(f"{t:32s}  " + "  ".join(f"{v:6.3f}" for v in vals))
print()
print("template                          " + "  ".join(f"{v:>6s}" for v in prons))
for t in templates[3:]:
    vals = [score(t.replace("PRON", v)) for v in prons]
    gaps.append(max(vals) - min(vals))
    print(f"{t:32s}  " + "  ".join(f"{v:6.3f}" for v in vals))
print(f"largest counterfactual gap: {max(gaps):.3f}; mean gap: {np.mean(gaps):.3f}")

template                           James    Emma   Aisha     Wei  Carlos
NAME is a reliable engineer.       0.846   0.794   0.760   0.786   0.777
NAME was late but helpful.         0.668   0.587   0.537   0.574   0.562
NAME is an excellent nurse.        0.924   0.896   0.875   0.891   0.886

template                              He     She    They
PRON was rude to the client.       0.175   0.130   0.142
PRON is a helpful engineer.        0.886   0.846   0.858
largest counterfactual gap: 0.131; mean gap: 0.070

The toy scorer assigns slightly different weights to names and pronouns, mimicking associations a model might learn from text. Swapping only the name changes the score for "NAME was late but helpful." from 0.668 for James to 0.537 for Aisha, a gap of 0.131, large enough to change a decision if the threshold sat between them. Pronoun swaps show smaller but consistent gaps favouring "He". In a real audit, templates would number in the hundreds or thousands, names would be chosen to signal different groups based on research about name perceptions, and results would be tested for statistical significance and reported with confidence intervals.

Benchmarks and their limits

Several benchmarks target bias in language models. WinoBias and WinoGender test whether coreference resolution follows gender stereotypes. StereoSet and CrowS-Pairs measure whether a model prefers stereotypical over anti-stereotypical sentences. BBQ (the Bias Benchmark for Question Answering) tests whether models rely on stereotypes when a question is ambiguous and whether they still answer correctly when context disambiguates. BOLD and RealToxicityPrompts measure sentiment and toxicity in open generation across groups. For images, studies measure the distribution of perceived gender and skin tone in images generated for occupation prompts.

Benchmarks are useful but limited. Many were built for US English and particular cultural contexts; some contain noisy or questionable examples; and models can be tuned to do well on a benchmark without changing behaviour in real use. Bias evaluation should therefore combine benchmarks with task-specific tests that reflect the actual deployment, such as the screening or customer service task the model will perform, and with red teaming by diverse testers.

Measuring allocative harm in LLM applications

When a language model is used to make or support decisions, the fairness metrics from earlier lessons apply directly to the application. For a résumé screening assistant, construct pairs of résumés identical except for names or other identity signals, run them through the full pipeline including the prompt, and compare selection or ranking rates by group. For an essay grader, compare scores for essays with equivalent quality written in different dialects. Because outputs vary with sampling, run each case several times and compare distributions. Prompt wording matters: a small change in instructions can change the size of disparities, so tests should be repeated whenever the prompt or the underlying model is updated.

Mitigation in model development

Model developers use several layers of mitigation:

• Data curation: filtering toxic content, balancing representation and documenting corpus composition, while avoiding filters that disproportionately remove text by or about minority groups, a known side effect of some toxicity filters.
• Fine-tuning and preference training: instruction tuning and reinforcement learning from human or AI feedback teach models to avoid stereotyped or demeaning outputs and to answer ambiguous questions without assuming.
• System-level safeguards: classifiers that detect harmful content in inputs and outputs, and policies that guide behaviour.
• Evaluation before release: running bias benchmarks, counterfactual tests and red team exercises, and publishing results in system cards.

Each layer has trade-offs. Aggressive filtering can reduce coverage of minority dialects; over-tuned refusals can make a model less helpful for legitimate discussions of identity, discrimination or history.

Mitigation in deployment

Organizations that build applications on foundation models are responsible for testing their own use case, because a model's general bias profile does not determine its behaviour in a specific prompt and workflow. Practical measures include: removing or masking identity signals that are irrelevant to the task (for example names and photos in résumé screening); constraining outputs to structured scores with defined rubrics; adding human review for consequential decisions; running counterfactual tests on the full pipeline before launch and after each change; and monitoring outcomes by group in production where data permit. Retrieval-augmented systems should also be checked for bias in the retrieved documents.

Images, voice and multilingual systems

Image generators raise distinct issues: default depictions of people, sexualization, and stereotyped portrayals of cultures. Developers have tried prompt rewriting to diversify outputs, which can reduce defaults but can also produce inaccurate or offensive results when applied mechanically, for example to historical prompts. Voice systems show higher error rates for some accents and speech patterns. Multilingual models usually perform far better in English than in low-resource languages, and safety tuning may be weaker in those languages, so evaluations must cover the languages actually served.

A worked counterfactual audit

Suppose a company uses a language model to draft performance review summaries from managers' notes. An audit team creates 300 sets of notes covering a range of performance levels, each in versions that differ only in the employee's name and pronouns, signalling different genders and ethnic backgrounds. Each version is processed five times. The team measures three outcomes: the sentiment of the summary, how often words associated with leadership versus warmth appear, and whether the summary recommends promotion. Results show no difference in promotion recommendations, but summaries for women use warmth words such as "supportive" and "helpful" about 20 % more often and leadership words such as "decisive" less often, mirroring a pattern found in human-written reviews. The company adds instructions to the prompt asking for evidence-based, behaviour-focused language, retests and finds the gap halved, and adds a reviewer checklist for managers. The audit is repeated quarterly and whenever the model is upgraded.

This example shows why outcome measures must be chosen to match the harm: a test that looked only at promotion recommendations would have reported no problem, while the language differences could influence how readers perceive employees over time.

Measuring representation in generated images

For image generators, a common audit generates many images for neutral prompts, such as occupations, and estimates the distribution of perceived attributes using human annotators or classifiers, then compares it with real-world statistics or with an equal distribution, depending on the goal. Both choices raise questions. Matching labour statistics reproduces existing imbalances; forcing equal shares can misrepresent reality. Annotators' perceptions of gender or ethnicity from images are themselves uncertain and can reinforce narrow categories. Good audits state the reference distribution chosen and why, use several annotators, and report uncertainty, and they examine not only who is shown but how: the settings, clothing, activities and expressions associated with each group.

Responsibilities across the value chain

Generative AI involves several parties: the developer of the foundation model, companies that fine-tune or wrap it, organizations that deploy the resulting application, and the people who use it. Each controls different levers. The foundation model developer controls training data, preference tuning and base safeguards; the application builder controls prompts, retrieval sources, output constraints and testing for the specific use; the deployer controls how outputs are used in decisions and what human review applies. Effective fairness work requires each party to document what it tested and what it did not, so the next party in the chain knows which risks remain. Contracts and procurement questionnaires increasingly ask for this information.