Training, validation and testing

With the appointment file cleaned, Jonah asked the obvious question: "Now do we train the model on all 410,000 rows?" Noor shook her head. "If we use every row to train, we have nothing left to tell us whether the model works on appointments it has never seen. That is the only thing we care about."

This lesson explains why data is split, what each part is for, how the split must respect time, and what happens after training, when the model is put to work through one of several kinds of inference.

Why split the data?

A model is useful only if it generalizes: if it performs well on new data, not just on the examples it learned from. Measuring a model on its own training data is like letting students grade the exam they wrote the answer key for. A very flexible model can memorize its training data and score almost perfectly there while being poor on anything new, which lesson 6 calls overfitting. Holding data back is how we find out.

The three sets

• The training set is the data the algorithm learns from. It is usually the largest part: commonly around 70 to 80 of every 100 examples.
• The validation set is used during development to compare models and tune their settings. When Noor tries a decision tree against a gradient-boosted model, or tries different tree depths, she compares them on validation data. Because choices are made by looking at it, the validation set slowly influences the model.
• The test set is kept aside until the very end and used once, to estimate how the chosen model will perform in real use. If people keep peeking at the test set and adjusting, it stops being a fair test.

The settings Noor tunes on validation data are called hyperparameters: choices made before training, such as how deep a decision tree may grow, how many trees to combine or how fast a neural network learns. They are different from parameters, the numbers the algorithm learns from the data during training, such as the weights inside a neural network. Searching for good hyperparameters is called hyperparameter tuning, and SageMaker AI can run that search automatically, as lesson 9 describes.

Splitting by time

How the rows are divided matters as much as the proportions. For many problems a random split is fine. For data where time matters, it is not. Appointment behaviour changes: a new reminder system, a pandemic, a clinic that moved. If the split is random, the model is trained on some appointments from the most recent month and tested on others from the same month, and it quietly benefits from knowing the future. In use, it will always be predicting a future it has not seen.

Noor used a time-based split. Appointments from the first 25 months trained the model, the next 5 months were for validation, and the last 6 months, ending in September 2026, were the test set. That mirrors how the model will be used: trained on the past, then asked about tomorrow.

A related idea is cross-validation: splitting the training data into several parts, training on all but one, checking on the one left out, and repeating so that every part takes a turn. It gives a more stable estimate when data is limited. For time series, the parts must still move forward in time.

What training actually does

During training, the algorithm repeatedly makes predictions on training examples, measures how wrong they are with a loss function, and adjusts the model's parameters to reduce that loss. For a neural network, each full pass through the training data is called an epoch, and the adjustment method is usually a form of gradient descent: nudging each weight a little in the direction that reduces the error. Training stops when the loss stops improving on validation data, or after a set budget of time or epochs. Large models can take days of computing time on specialized hardware; the no-show model, a gradient-boosted tree on tabular data, trained in a few minutes.

Inference: putting the model to work

Once a model is trained and tested, it is deployed so that applications can use it. The exam guide names four ways of running inference, and choosing between them is a common question.

For the no-show pilot, Noor chose batch inference: at 6 p.m. each day, all of the next day's bookings are scored and a list goes to each clinic's front desk. Nobody needs an answer in milliseconds, and batch is the cheapest option because no endpoint runs all day. If BookVet later wants to show a score while a client books online, a real-time or serverless endpoint would be needed. Amazon SageMaker AI supports all four kinds, which lesson 9 covers: batch transform jobs, real-time endpoints, asynchronous inference and serverless inference.

# A time-ordered split versus a random split, on monthly miss rates.
months = list(range(1, 37))               # 36 months, oldest first
# The miss rate fell after a new reminder system started in month 25.
miss_rate = [0.09 if m < 25 else 0.06 for m in months]

train = months[:25]; valid = months[25:30]; test = months[30:]
def avg(ms): return round(sum(miss_rate[m - 1] for m in ms) / len(ms), 3)

print("train months", train[0], "-", train[-1], "avg miss rate", avg(train))
print("valid months", valid[0], "-", valid[-1], "avg miss rate", avg(valid))
print("test  months", test[0], "-", test[-1], "avg miss rate", avg(test))
print("a model that just learned the training average would overestimate misses in the test period")

train months 1 - 25 avg miss rate 0.089
valid months 26 - 30 avg miss rate 0.06
test  months 31 - 36 avg miss rate 0.06
a model that just learned the training average would overestimate misses in the test period

The output shows why the split order matters. The miss rate dropped after the new reminder system, so the test period behaves differently from most of the training period. A time-based split reveals that shift during development; a random split would have hidden it by mixing old and new months into every set.

Training on AWS, briefly

Where does the computing for training happen? For small tabular models, a laptop or a single cloud instance is enough. For larger work, Amazon SageMaker AI starts training jobs that run on instances chosen for the task, read training data from Amazon S3, write the trained model back to S3, and shut down when finished, so you pay only while training runs. Deep learning jobs often use instances with GPUs or AWS's own machine learning chips, AWS Trainium for training and AWS Inferentia for inference. Because the job is defined in code, the same training can be repeated exactly later, which matters for the MLOps practices in lesson 8.

Noor ran the first experiments in SageMaker Canvas, a no-code interface, so Jonah could follow along, then repeated the best configuration as a SageMaker AI training job so that it could be rerun on a schedule. The data never left the Cascade Ridge AWS account, and the training job used an IAM role that could read only the prepared bucket, following the least-privilege habits from AWS-102.

Latency, throughput and cost

Three measures decide which inference type is right. Latency is how long one prediction takes to return; a client waiting on a booking screen notices anything over a second or two. Throughput is how many predictions the system can make in a period; the evening batch must score about 2,600 bookings, which a single small instance does in minutes. Cost depends on how capacity is billed: an always-on endpoint costs money every hour whether it is busy or idle, a batch job costs only while it runs, and serverless costs per request. A useful habit is to write down the latency each use really needs before choosing. Kwame initially asked for real-time scores "to be safe", but when Noor asked who would look at a score within a second of the booking, the answer was nobody: the front desk only uses the list the next morning.

Where the model runs also matters for privacy. All inference for the pilot runs in the company's own AWS account in ca-central-1, so client booking data stays in Canada, matching the Region choices made in AWS-101. That would be harder to guarantee with a tool that sent data to an outside service, which is a question to ask of any AI product.

Inference at the edge

Some models run not in a Region but on a device close to where data is produced, such as a camera, a phone or an ultrasound machine. Edge inference avoids network delays and keeps data on site, at the cost of smaller, simpler models and more work to update them. The radiograph-quality idea might eventually run beside the X-ray machine for that reason, but the first experiment would run in the cloud, where it is easier to measure and change.

Checking predictions after deployment

Testing does not end when the model goes live. For the no-show model, the true outcome of each prediction is known the next day, so recall and precision can be recalculated every week on real appointments, a form of ongoing testing on data the model has never seen. Some models never get such quick answers: a model that predicts which pets will develop a chronic condition might wait years. In those cases teams rely more on monitoring inputs for change and on periodic reviews by experts. Knowing in advance how and when true outcomes will arrive is part of designing any ML system, and lesson 8 builds monitoring on it.

Noor summarized the lesson for Jonah in three sentences he could repeat at the exam. Keep data the model never sees, and look at it only at the end. Split by time whenever time matters. Choose the inference type from how the answers will be used, not from what sounds most impressive.