MLOps concepts

Gwen Halvorsen, the cloud architect running the AWS move, joined the AI hour on October 27. She had a question Noor had been expecting. "Your model lives in a notebook on your laptop. When you leave in June, who retrains it, and how do they know it still works?" Noor smiled. "That question has a name. It is MLOps."

Machine learning operations (MLOps) applies the habits of good software operations, automation, version control, testing and monitoring, to machine learning. Its goal is to take models from experiment to production reliably and keep them working there. This lesson covers the MLOps concepts the exam guide lists: experimentation, repeatable processes, scalable systems, technical debt, production readiness, model monitoring and retraining.

Experimentation, tracked

Data science involves many experiments: different features, algorithms and hyperparameters. MLOps does not discourage experimenting; it insists on recording it. For every training run, a team records the data version, the code version, the settings, the metrics and the resulting model artifact. SageMaker Experiments and similar tools do this automatically. Noor had run 34 experiments by late October; with tracking, she could say exactly which data and settings produced the model with 0.62 recall, and anyone could rerun it.

Repeatable processes

A model built by hand, through a sequence of notebook cells run in the right order by one person, is fragile. MLOps turns the lifecycle into an automated pipeline: defined steps for data preparation, training, evaluation and registration, run the same way every time, triggered on a schedule or by new data. SageMaker Pipelines provides this on AWS. Repeatability has three benefits. Results can be reproduced and audited. Retraining becomes routine instead of a project. And a new team member can understand the process by reading it.

Versioning applies to everything, not just code: the training data snapshot, the features, the model and its configuration. A model registry, such as the SageMaker Model Registry, stores model versions with their metrics and an approval status, so that the deployment step always uses a version someone approved, and an older version can be restored quickly if a new one misbehaves.

Scalable systems

A process that works for one model on one laptop may not work for ten models on growing data. MLOps favors infrastructure that scales without rewriting: training jobs that run on managed instances sized to the task, data stored in Amazon S3 rather than on a personal drive, and endpoints that scale with traffic. The same infrastructure-as-code habits used for the rest of Cascade Ridge's AWS accounts, covered in AWS-102 and AWS-104, apply to ML resources too.

Managing technical debt

Technical debt is the future cost of shortcuts taken today. ML systems accumulate it quickly and in unusual ways: undocumented features nobody remembers creating, pipelines that depend on a column in a source system that might be renamed, models that silently rely on another model's output, and glue code that grows around a small model until it is larger than the model itself. MLOps practices reduce this debt: documenting features and their sources, testing data as well as code, removing unused features, and keeping each model's dependencies explicit. Noor kept a short design record for the no-show model listing every feature, its source and why it was included.

Production readiness

Before a model goes live, a team should be able to answer a checklist. Noor and Gwen agreed on these questions for Cascade Ridge.

• Does the model beat the agreed baseline on the test set, and have the business owner and the clinical lead reviewed it?
• Is the pipeline automated, versioned and reproducible from source control?
• Are inputs validated, so that a broken data feed stops the job instead of producing nonsense scores?
• Is access controlled with least-privilege IAM roles, and is the data encrypted, as AWS-102 described?
• Is there monitoring with alerts, a named owner, and a tested way to roll back to the previous version?
• Is there a fallback if the model is unavailable? For the no-show list, the fallback is the old rule: an extra reminder for anyone booked more than three weeks ahead.

Model monitoring

A deployed model can degrade without any error message. Monitoring watches for several kinds of change.

• Data drift: the distribution of inputs changes. After BookVet added online booking, far more appointments were booked late at night and weeks ahead, which the model had seen less often.
• Concept drift: the relationship between inputs and the outcome changes. If a new text-reminder system makes long-advance bookings much more reliable, the old link between days booked ahead and misses weakens.
• Model quality: the model's actual accuracy, measured once true outcomes are known. For the no-show model, the answer arrives the next day, so recall and precision can be tracked weekly.
• Bias drift: changes in how performance differs between groups, which SageMaker Clarify can watch alongside Model Monitor.

SageMaker Model Monitor compares live data and predictions with a baseline captured at training time and raises alerts through Amazon CloudWatch when they diverge beyond set limits.

Retraining and safe rollout

When monitoring shows a decline, or on a regular schedule, the model is retrained on fresher data through the same pipeline, evaluated against the current version, and promoted only if it is better. New versions can be introduced carefully: a shadow deployment runs the new model alongside the old one without using its output, so the two can be compared on real traffic; a canary or A/B test sends a small share of traffic to the new version first. For a nightly batch job, the simplest safe pattern is to score with both versions for two weeks and compare results before switching.

# A simple data-drift check: compare the share of bookings made far in advance.
def share_far_ahead(days_list, cutoff=21):
    return sum(d >= cutoff for d in days_list) / len(days_list)

training_sample = [2, 5, 7, 14, 3, 30, 9, 1, 6, 12, 25, 4, 8, 10, 2, 3, 15, 7, 6, 28]
this_week       = [21, 35, 2, 28, 40, 7, 22, 30, 3, 25, 14, 45, 6, 21, 33, 9, 27, 1, 24, 30]

base, now = share_far_ahead(training_sample), share_far_ahead(this_week)
print("far-ahead share at training:", base, "| this week:", now)
if abs(now - base) > 0.2:
    print("ALERT: input distribution has drifted; review model quality and consider retraining")

far-ahead share at training: 0.15 | this week: 0.65
ALERT: input distribution has drifted; review model quality and consider retraining

Real monitoring uses statistical tests across every feature, but the principle is the same: capture a baseline at training time, compare live data with it on a schedule, and alert a named person when the difference is too large. Here the share of far-ahead bookings jumped after online booking launched, which is exactly the kind of change that should prompt a closer look.

Testing in ML

Software teams test code; ML teams must also test data and models. Noor set up three kinds of automatic check in the pipeline. Data tests confirm that each nightly input looks sane: the expected columns are present, no clinic code is unknown, days booked ahead is never negative, and the number of bookings is within a normal range for that weekday. Model tests confirm that a newly trained model beats the baseline rule and the current production version on the same test data, and that its recall at each clinic is within an agreed band. Behavior tests run a handful of hand-made bookings through the model to check it responds sensibly, for example that a booking made the same day is scored as lower risk than an otherwise identical one made a month ahead. If any test fails, the pipeline stops and alerts its owner instead of promoting a broken model. These checks are cheap to run and catch the quiet failures that would otherwise reach the front desk.

MLOps for foundation models

The same ideas apply when the model is a foundation model used through an API, although the details shift. There may be no training pipeline to version, but there are prompts, retrieval data and model versions to track; a provider's model update can change outputs just as drift can. Teams sometimes call this LLMOps or operations for generative AI. Monitoring focuses on output quality, safety and cost per request rather than recall on labeled outcomes, and evaluation often mixes automated checks with human review. AWS-113 covers these practices for applications built on Amazon Bedrock. For Jonah, the lesson was that MLOps is not tied to one kind of model: any AI system used in production needs versioning, testing, monitoring, an owner and a way back.

How much MLOps is enough?

A small company does not need the tooling of a large technology firm. Noor scaled the practices to Cascade Ridge: one pipeline, one registry, weekly monitoring, a quarterly retraining schedule and a single page of documentation. The essential test is simple. Could someone other than the original builder retrain, check and, if necessary, roll back the model next year, using only what is written down and automated? If the answer is yes, the MLOps is sufficient for now.

Retraining schedules and triggers

There are two common ways to decide when to retrain. Scheduled retraining runs on a fixed calendar, such as every quarter, which is simple and predictable. Triggered retraining runs when monitoring shows that drift or quality has crossed a limit. Many teams combine them: a regular schedule as a safety net and triggers for sudden changes. For Cascade Ridge, quarterly retraining keeps the model current with seasonal patterns, and an alert on weekly recall falling below 0.55 would prompt an earlier run. Each retrained model must still pass the same tests and approvals as the first one; automation speeds up the work but does not remove the checks.

Who is responsible in production?

MLOps also answers an organizational question: when something goes wrong at 6 p.m. on a Friday, who is called? Gwen insisted that the no-show pipeline have an on-call owner listed in the same place as the owners of every other production system, with a short runbook: how to tell whether the job ran, how to switch to the fallback rule, how to roll back to the previous model version, and whom to tell. The runbook was tested once in December by deliberately breaking the input file. The fallback worked, the alert reached Noor's phone, and the front desks received a list based on the old rule without noticing anything had happened. That rehearsal, Gwen said, was worth more than any diagram.