Transparency and explainability

A woman named Dalia Haddad has been turned down for a small business loan at Lakeside Credit Union. The letter says her application 'did not meet our lending criteria' and invites her to reapply in a year. She runs a catering business, has never missed a rent payment and does not understand what went wrong. She writes to the community group that raised concerns about the model, and the group forwards her letter to Kwame. 'She's asking the most basic question there is,' Kwame says at the next session. 'Why? And I'm not sure anyone at the credit union can tell her.'

This lesson is about the demand to understand automated decisions. It asks what transparency and explainability mean, why they matter, what kinds of explanation are possible, and what the law requires. It also asks whether some systems are too opaque to be used for some decisions at all.

Kinds of opacity

In 2016 the sociologist Jenna Burrell distinguished three reasons a machine learning system may be opaque. The first is intentional secrecy: companies treat models and data as trade secrets, and governments may withhold details to prevent people from gaming the system. The second is technical illiteracy: even if the code were published, most people could not read it. The third is opacity that comes from the way the system works: models with millions or billions of learned parameters find patterns that do not correspond to human concepts, so that even their developers cannot say in ordinary terms why a particular output was produced.

The three kinds call for different responses. Secrecy can be addressed by law, contracts and auditing. Illiteracy can be addressed by explanations designed for the people who need them. Inherent complexity is the hardest: it may require new technical methods, simpler models, or limits on where complex models are used.

Why explanations matter

Explanations serve several different purposes, and the purpose shapes what kind of explanation is needed.

• Contestability: a person affected by a decision needs enough understanding to challenge it, correct errors in their data or show that their situation is different.
• Recourse: a person denied a benefit wants to know what they could change to get a different result.
• Accountability: regulators, courts and auditors need to know how a system works to judge whether it is lawful and fair.
• Debugging and safety: developers need to understand models to find errors, biases and failures.
• Appropriate trust: users such as nurses and loan officers need to know when to rely on a system and when to override it.
• Respect: some philosophers argue that people are owed reasons for decisions that significantly affect them, as part of being treated as rational agents rather than objects to be sorted.

The last point connects to deontology. Kantian ethics treats persons as beings who act for reasons and can evaluate reasons given to them. A decision that affects someone deeply but can be given no reason they could understand treats them, in a sense, as a thing to be managed. This idea underlies the long tradition in administrative law that public authorities must give reasons for their decisions. Critics reply that people accept unexplained decisions all the time, from doctors' intuitions to judges' discretion, and that what matters is accuracy and fairness, not explanation. Varga asks the class whether they would rather have a more accurate system they cannot understand or a less accurate one they can. The class splits, and the split depends on the stakes and on whether there is another way to check the system.

Types of explanation

Explanations of machine learning models are often classified along two dimensions. A global explanation describes how the model behaves in general: which features matter most, how outputs change as inputs change. A local explanation describes why the model produced a particular output for a particular case, such as Dalia's application. Explanations can also be intrinsic, when the model is simple enough to be understood directly, or post hoc, when a separate method is used to approximate or probe a complex model after it is built.

Interpretable models include short decision trees, rule lists, and linear or logistic models with a modest number of features, where the contribution of each feature can be read directly. For a linear score, the contribution of each feature is its weight times its value, and the contributions add up to the score. That makes local explanations straightforward.

# A simple, interpretable credit score for Dalia's application (illustrative weights)
weights = {"years_in_business": 6.0, "missed_payments_24m": -18.0,
           "debt_to_income": -40.0, "credit_history_years": 3.0, "revenue_growth": 20.0}
intercept = 30.0
threshold = 46.0
dalia = {"years_in_business": 3, "missed_payments_24m": 0,
         "debt_to_income": 0.45, "credit_history_years": 2, "revenue_growth": 0.30}

contrib = {k: weights[k] * dalia[k] for k in weights}
score = intercept + sum(contrib.values())
print("score:", round(score, 1), "threshold:", threshold, "approved:", score >= threshold)
for k, v in sorted(contrib.items(), key=lambda kv: kv[1]):
    print(f"  {k:22s} {v + 0.0:+.1f}")
# Counterfactual: how far would debt-to-income need to fall to reach the threshold?
needed = (threshold - (score - contrib["debt_to_income"])) / weights["debt_to_income"]
print("debt-to-income needed for approval:", round(needed, 2))

score: 42.0 threshold: 46.0 approved: False
  debt_to_income         -18.0
  missed_payments_24m    +0.0
  credit_history_years   +6.0
  revenue_growth         +6.0
  years_in_business      +18.0
debt-to-income needed for approval: 0.35

For a model like this, the explanation for Dalia is clear: her debt-to-income ratio pulled the score down more than anything else, and her short credit history added little. The counterfactual is also clear: if her ratio fell from 0.45 to about 0.35, she would be approved. That is an explanation she can understand, contest if the debt figure is wrong, and act on.

For complex models, post hoc methods try to provide similar information. LIME, introduced in 2016, explains an individual prediction by fitting a simple model to the complex model's behavior in the neighborhood of that case. SHAP, introduced in 2017 and based on Shapley values from cooperative game theory, assigns each feature a share of the difference between a prediction and an average prediction. Counterfactual explanations, proposed by Sandra Wachter, Brent Mittelstadt and Chris Russell in 2017, tell a person the smallest change to their situation that would have produced a different outcome, without trying to describe the model's internal logic. Saliency maps highlight the parts of an image that most influenced a classification.

The limits of post hoc explanation

Post hoc explanations have serious limits. They are approximations, and different methods can give different explanations for the same prediction. They can be unstable, changing noticeably with small changes in the input. They can be manipulated: researchers have shown that a model can be built to behave in a biased way while producing explanations that hide the bias. And an explanation that is technically accurate can still be useless to the person who receives it, if it lists dozens of features with numerical weights.

In 2019 the computer scientist Cynthia Rudin argued that, for high-stakes decisions, people should stop trying to explain black-box models and use interpretable models instead. She pointed to evidence that in many domains with structured data, such as credit and criminal justice, well-designed interpretable models perform about as well as complex ones. If that is true, the supposed trade-off between accuracy and explainability is often illusory, and choosing a black box imposes the costs of opacity for little gain. Others reply that in domains such as medical imaging and language, complex models do perform much better, and that the right response there is rigorous testing and monitoring rather than a ban.

Kwame says his team chose a complex model for one product because it scored slightly better on a benchmark, and no one asked whether the difference mattered. Varga says that is a common pattern: the accuracy gain is measured and the opacity cost is not.

Explanations for the people who use the system

Explanations are not only for the people a system decides about. They are also for the people who work alongside it, such as Raúl and his colleagues if Riverton General adopts Pathfinder. Research on automation bias shows that people tend to accept a system's output when it is presented confidently, and to overlook errors they would have caught on their own. Explanations can help or hurt. A good explanation, such as showing that a patient was ranked high mainly because of a low oxygen reading, lets a nurse check the reasoning against what she sees. A poor explanation, such as a confidence score with no reasons, can increase deference without increasing understanding. Studies have found that explanations sometimes make people trust a system more even when it is wrong. Raúl says the explanation he would want is simple: the three things that drove the ranking, and a warning when the model is outside the kinds of cases it was trained on.

Transparency beyond the model

Explaining individual decisions is only one kind of transparency. Others can matter as much.

• Transparency about use: telling people when an automated system is being used to make or support decisions about them, and when they are interacting with an AI rather than a person.
• Process transparency: publishing how a system was designed, what data it was trained on, how it was tested, what its known limitations are and who is responsible for it. Model cards and datasheets are tools for this.
• Outcome transparency: publishing aggregate results, such as error rates and outcomes by group, so that the public can judge whether the system works and is fair.
• Access for auditors: allowing independent researchers or regulators to examine the system and its data under confidentiality.

Transparency also has costs and limits. Full disclosure of how a fraud-detection or tax-audit model works may help people evade it. Publishing training data may expose personal information. And transparency without power does little: knowing why you were denied a loan helps only if you can appeal, correct errors or go elsewhere. Some scholars argue that the real goal is accountability, and that transparency is one means to it among others.

What the law requires in the United States

In credit, the Equal Credit Opportunity Act and its implementing Regulation B require lenders to give applicants who are denied credit a statement of the specific principal reasons for the denial, and the Fair Credit Reporting Act requires notice when a credit report contributed to an adverse decision. In 2022, the Consumer Financial Protection Bureau issued a circular stating that these requirements apply in full when lenders use complex algorithms: a lender cannot avoid giving specific reasons because its model is too complicated to explain. Due process under the Fifth and Fourteenth Amendments requires notice and an opportunity to be heard before government deprives people of significant interests, and courts have heard challenges to opaque government algorithms; in State v. Loomis (2016), the Wisconsin Supreme Court allowed a proprietary risk score to be used at sentencing but required warnings about its limitations. Some state and local laws, such as New York City's Local Law 144 on automated employment decision tools, require notice and published bias audits.

Outside North America, the European Union's General Data Protection Regulation, in force since 2018, gives people the right not to be subject to decisions based solely on automated processing that significantly affect them, with exceptions, and requires that they receive meaningful information about the logic involved. Scholars have debated whether this amounts to a right to explanation. The EU's AI Act, adopted in 2024, adds transparency duties, including that people be told when they are interacting with an AI system and that some AI-generated content be labeled.