Limitations: hallucination, bias and data quality

In mid-December the review summarizer produces a digest for the Summit 30 sleeping bag that says customers "consistently praise its warmth below minus 20". Jules almost forwards it to the buying team before checking the reviews. Several reviewers had praised the bag, but none had used it below minus 12, and two had complained that it was cold at minus 15. The model had blended praise for warmth with the bag's marketing rating and produced a confident claim no reviewer made. Nobody was harmed, but the near miss becomes the starting point for this lesson on the limitations of generative AI and how to manage them.

Why limitations are a leadership topic

Every technology has limits, but generative AI's limits are unusual because its failures look like successes. A wrong answer arrives in the same fluent, confident prose as a right one. A biased summary reads as balanced. For that reason the Generative AI Leader exam spends real attention on limitations: leaders decide where generative AI is used, what checks surround it and who is accountable when it errs. The exam guide lists the common limitations as data dependency, knowledge cutoff, bias and fairness, hallucinations and edge cases.

Hallucination

A hallucination is output that is fluent and plausible but false, unsupported or nonsensical: an invented fact, a fake citation, a policy that does not exist, or, as with the sleeping bag, a claim stitched together from fragments. Hallucinations happen because a language model generates likely-sounding text rather than looking facts up. When it lacks information, or the information is ambiguous, it still produces the most probable continuation, which can be wrong.

Hallucinations are more likely when a question asks for specific facts the model was never given, such as exact numbers, dates, names and quotations; when a prompt pressures the model to answer rather than admit uncertainty; when inputs are long and the relevant detail is buried; and when temperature is high. They can never be eliminated completely, but they can be reduced a great deal.

After the sleeping-bag incident, Ravi changes the summarizer in three ways. Each claim in a digest must quote or cite the reviews it came from. Any temperature or numerical claim is checked automatically against the review text. And digests are labelled as AI-generated drafts until a buyer has reviewed them.

Knowledge cutoff

A model's knowledge comes from training data collected up to a certain date, its knowledge cutoff or training cutoff. It knows nothing of events after that date and may present outdated information as current: last year's prices, a superseded regulation, a product that has been discontinued. Models often do not know their own cutoff precisely, and may not realize that time has passed.

The remedy is grounding with current information: Birchline's live catalogue for products, the current policy documents for rules, and grounding with Google Search for public events. For anything time-sensitive, the application should supply the current date and the relevant facts rather than relying on what the model remembers.

Data dependency and data quality

Generative AI is only as good as the data it learned from and the data it is given. This data dependency shows up in two places. The model's training data shapes its general knowledge, style and blind spots. The organization's own data, used for grounding and tuning, shapes its answers about the business. Errors, gaps and inconsistencies in either flow straight into output, delivered fluently.

Ravi's data quality checklist from lesson 1 becomes concrete. Accuracy: are the product specifications correct? Completeness: do all products have specifications, or are some missing, inviting the model to fill gaps? Consistency: is weight always in grams, or sometimes in ounces? Timeliness: are discontinued products marked? Relevance: does the help desk knowledge base include articles about a loyalty scheme that ended in 2023? Accessibility and permission: can the system reach the data, and is it allowed to? When the team audits the catalogue, they find 340 products with no weight recorded and 90 with weights in mixed units, problems that would have produced confidently wrong descriptions.

# A quick data-quality audit of product rows before using them for grounding.
products = [
    {"sku": "TN-201", "weight": "1840 g", "status": "active"},
    {"sku": "TN-202", "weight": "", "status": "active"},
    {"sku": "SB-030", "weight": "3.4 lb", "status": "active"},
    {"sku": "JK-110", "weight": "620 g", "status": "discontinued"},
    {"sku": "BT-075", "weight": "1150 g", "status": "active"},
]
missing = [r["sku"] for r in products if not r["weight"]]
non_metric = [r["sku"] for r in products if r["weight"] and not r["weight"].endswith(" g")]
retired = [r["sku"] for r in products if r["status"] != "active"]
print("missing weight:", missing)
print("weight not in grams:", non_metric)
print("exclude from shopper answers:", retired)

missing weight: ['TN-202']
weight not in grams: ['SB-030']
exclude from shopper answers: ['JK-110']

Bias and fairness

Bias in AI means systematic patterns in output that unfairly favour or disadvantage people or groups, or that misrepresent them. Models learn from large amounts of human-produced data, which contains the stereotypes, imbalances and gaps of the societies that produced it. A model may associate certain jobs with one gender, describe some communities in more negative terms, perform worse on dialects or languages that were rare in its training data, or generate images that default to a narrow idea of who goes camping.

Fairness is the goal of outcomes that do not unjustly differ across groups. What counts as fair depends on context and values, so fairness requires human judgement, not just a metric. For Birchline, fairness questions arise in several places: whether the shopping assistant gives equally helpful answers to questions written in French and in English, or in plain and in formal language; whether generated campaign images represent the full range of Birchline's customers; and, most seriously, whether any AI system ever influences decisions about people, such as screening job applicants, which Birchline's draft policy forbids without specific review.

Managing bias involves testing outputs across groups and languages, reviewing training and grounding data for imbalance, writing prompts that ask for inclusive output, involving diverse reviewers, and keeping people accountable for decisions that affect individuals. Lesson 11 places these practices within responsible AI governance.

Edge cases

An edge case is an unusual input that a system was not designed or tested for: a question in an unexpected language, a product with no reviews, a customer who asks about an item bought in another country, a sarcastic review the model reads as praise, or a request that mixes three topics. Generative models often handle common cases well and fail unpredictably on rare ones, precisely because those cases were rare in their data and in the evaluation set.

The defences are to look for edge cases deliberately and add them to the evaluation set, to design fallbacks ("I'm not sure; let me connect you with someone"), to monitor for unusual inputs in production, and to make handoff to a person easy. Jules's test plan now includes a section of strange questions, contributed by the help desk team, who see the oddest requests every day.

Other limitations leaders should know

Several further limitations shape design decisions.

• Non-determinism: the same prompt can produce different outputs, so processes must not depend on identical wording (lesson 4).
• Reasoning limits: models can make errors in arithmetic, logic and multi-step planning, especially without step-by-step prompting or tools. Calculations such as refunds should be done in code.
• Context limits: very long inputs can cause models to overlook details, and everything must fit in the context window.
• Security weaknesses: prompt injection and attempts to extract system instructions or data, which screening and least privilege address.
• Privacy risks: prompts and outputs can contain personal information, which must be protected and must not be used carelessly for tuning.
• Intellectual property questions: generated content may resemble existing works, and rules about ownership and use are still developing.
• Cost and latency: large models and long prompts are slower and more expensive, which limits where they make sense.

Human in the loop

The practice that cuts across every limitation is human in the loop: designing processes so that people review, approve or correct AI output where the stakes justify it. The level of human involvement should match the risk. A typo in an internal email summary is low stakes; a wrong statement in a customer-facing policy answer is medium; a decision affecting someone's job, health, money or legal rights is high and needs a person to decide.

Human review has its own pitfall: automation bias, the tendency of people to accept a system's output without real scrutiny, especially when it is usually right. Reviewers need time, clear criteria and the authority to reject output, and their approval rate should be watched; a reviewer who approves everything may not be reviewing.

A limitations register

Teresa asks each project to keep a short limitations register: a list of the known ways the system can fail, how likely and serious each is, the controls in place and who owns each one. The policy assistant's register includes hallucinated policies, outdated documents, questions in languages other than English and French, and attempts to extract the system instructions. The register is reviewed before launch and whenever an incident or evaluation reveals a new failure. It turns limitations from vague worries into managed risks, and it gives reviewers, auditors and the board a clear picture of what the team knows and is doing about it.