Bias, variance and overfitting

Kwame, curious, had trained his own no-show model over a weekend: a very deep decision tree with no limits on how far it could grow. On the training data it was nearly perfect, catching 99 of every 100 misses. On the validation data its recall fell to 0.48, worse than Noor's simpler model. "It learned the training data by heart," Noor said. "Including all the accidents."

This lesson explains what happened. It covers the two kinds of error every model balances, bias and variance, the problems they cause, underfitting and overfitting, and the practical ways to find the balance.

Two sources of error

In statistics, bias is error from wrong or overly simple assumptions. A model with high bias misses real relationships in the data, so it is consistently wrong in the same direction. Predicting every visit takes 30 minutes, regardless of how many procedures are booked, is a high-bias model: it ignores an obvious pattern.

Variance is error from being too sensitive to the particular training data. A model with high variance changes a lot when trained on a slightly different sample, because it has learned quirks and coincidences, not just the underlying pattern. Kwame's unlimited tree had high variance: if a client with a particular combination of visit type, weekday and clinic missed once by chance, the tree created a rule just for them.

This statistical bias is not the same as the fairness meaning of bias in lesson 1, though the words overlap. A model can have low statistical bias overall and still treat a group unfairly, and AWS-114 deals with that kind. When the exam says "bias and variance" together, it means the statistical kind described here.

Underfitting and overfitting

These two kinds of error produce two recognizable problems.

• Underfitting happens when a model is too simple to capture the pattern. It performs poorly on training data and on new data alike. The symptom is low performance everywhere. It is a high-bias problem.
• Overfitting happens when a model learns the training data too closely, including its noise. It performs very well on training data and noticeably worse on new data. The symptom is a large gap between training and validation scores. It is a high-variance problem.

A good fit sits between them: the model captures the real relationships and performs nearly as well on validation data as on training data. Reducing bias usually means adding flexibility, which tends to increase variance, and reducing variance usually means constraining the model, which tends to increase bias. This tension is the bias-variance trade-off, and most of the practical work of training a model is finding the right place on it.

Fixing underfitting

If a model underfits, give it more to work with. Add features that carry real information: Noor's first test model used only days booked ahead and underfit badly; adding past miss count and reminder confirmation improved it sharply. Use a more flexible algorithm, such as moving from a straight line to a tree-based model. For neural networks, train longer or use a larger network. Removing excessive regularization, described next, also helps.

Fixing overfitting

Overfitting has more remedies, and the exam likes to ask about them.

• Get more training data. More examples make it harder for the model to memorize coincidences. When real data is limited, data augmentation creates varied copies, such as slightly rotated or brightened radiographs for an image model.
• Simplify the model. Limit the depth of a tree, use fewer features, or choose a smaller network. Noor capped her trees at a depth of six.
• Regularization. Add a penalty during training for complexity, such as large weights, so that the model prefers simpler explanations. Dropout, which randomly ignores some neurons during each training step, is a common regularization method for neural networks.
• Early stopping. Watch the validation score during training and stop when it stops improving, even if the training score could still rise.
• Ensembles. Combine many models, as a random forest combines many trees trained on different samples. Averaging smooths out the quirks of individual models. The gradient-boosted model Noor used is also an ensemble, built by adding small trees one at a time.
• Remove leaky or noisy features. Features that encode chance or leak the answer encourage memorizing.

Fit changes over time

Even a well-fitted model can drift into a poor fit later, without any change to the model itself. If client behavior changes, because a new online booking flow makes rebooking easier, for example, the patterns the model learned stop matching reality. This is called drift, and it is why lesson 8 treats monitoring and retraining as part of the job, not an afterthought.

# Memorizing versus learning a simple pattern for visit length.
train = [(1, 22), (1, 18), (2, 31), (2, 29), (3, 41), (3, 38), (1, 35)]  # (procedures, minutes); last row is unusual
test = [(1, 21), (2, 30), (3, 40), (1, 19)]

# Model A: memorize the exact training answer for each input (last one seen wins).
memo = {}
for p, m in train:
    memo[p] = m
# Model B: average minutes per procedure count (a simple, smoother pattern).
groups = {}
for p, m in train:
    groups.setdefault(p, []).append(m)
avg = {p: sum(v) / len(v) for p, v in groups.items()}

def mae(model, rows): return round(sum(abs(model[p] - m) for p, m in rows) / len(rows), 1)
print("memorize: train MAE", mae(memo, train), "| test MAE", mae(memo, test))
print("average : train MAE", mae(avg, train), "| test MAE", mae(avg, test))

memorize: train MAE 5.0 | test MAE 8.2
average : train MAE 3.6 | test MAE 2.6

The memorizing model keeps the last answer it saw for each input. For single-procedure visits, that is the one unusual 35-minute visit, and it carries that quirk into every prediction, so its test error is roughly three times the averaging model's. A real memorizing model, such as an unlimited tree, can go further and fit every training row exactly, which makes its training score look perfect while its test score suffers in the same way. That gap between training and test error is the signature of overfitting, and it is exactly what Kwame saw with his unlimited tree.

Learning curves

A useful picture for diagnosing fit is a learning curve: training and validation scores plotted against the amount of training data. For an overfitting model, the training score starts very high and the validation score low, and the gap narrows slowly as data is added; more data is likely to help. For an underfitting model, both curves flatten out close together at a disappointing level; adding data will not help, and the model needs more capacity or better features. Noor drew the curve for her no-show model by training on 50,000, 100,000, 200,000 and all 287,000 training rows. The validation recall rose from 0.55 to 0.62 and was still creeping upward, which suggested that another year of history, once available, might help a little more.

For neural networks a similar picture over training time, loss against epochs, shows where early stopping should happen: the training loss keeps falling, while the validation loss falls, flattens and then begins to rise as the network starts memorizing. The best moment to stop is just before that rise. SageMaker AI training jobs can log both curves so that they can be inspected after each run.

Bias, variance and the kind of model

Different model families sit at different places on the trade-off by default. A straight-line regression has high bias and low variance: it cannot bend to fit complicated patterns, but it gives similar results on any reasonable sample. A deep, unconstrained decision tree or a large neural network has low bias and high variance: it can fit almost anything, including noise. Ensembles such as random forests reduce variance by averaging many trees, and gradient boosting reduces bias by adding trees that correct earlier mistakes, with settings such as tree depth and learning rate controlling how far it goes. Part of choosing a model is choosing a starting point on the trade-off that suits the amount and kind of data available. With 287,000 training rows and about 20 features, a moderately constrained gradient-boosted model was a sensible place for Noor to start.

Foundation models face the same tension in a different form. When a large pre-trained model is fine-tuned on a small company data set, it can overfit to those examples and lose some of its general ability. AWS-113 returns to this when it covers fine-tuning and evaluation.

Spotting overfitting in practice

Gaps between training and validation scores are the main signal, but there are others. A model whose results change a lot when it is retrained on a slightly different sample is showing high variance. Rules that look oddly specific when explained, such as "Thursday bookings at one clinic for a particular visit type are very risky", often reflect a handful of coincidences. Extremely high scores should also prompt suspicion before celebration: a recall of 0.99 on a hard problem is more often a sign of leakage or overfitting than of a brilliant model. Noor's habit was to look at the top features and a sample of individual predictions with Andre after every promising result, asking whether each made sense to someone who knows the clinics. Kwame's unlimited tree failed that test quickly: its most important rule mentioned a single booking clerk's initials, a field that should never have been in the data.

Underfitting has its own signals: predictions that barely vary between very different cases, or a model that does no better than the baseline of predicting the average or the most common class. Both problems are easier to spot when the baseline is always reported beside the model's scores, which is one more reason lesson 2 insisted on it.

The practical routine Noor taught Jonah fits in four steps. Always record training and validation scores side by side. If both are poor, add information or flexibility. If training is far better than validation, add data or constraints. And when the two are close and good, stop adjusting, test once on held-out data, and move on to deployment and monitoring. Most of the remaining art lies in recognizing which of those situations you are in, which the exam tests with short scenarios exactly like the three models in the worked example above.