The machine learning development lifecycle

By late October, Jonah had seen data cleaned, a model trained and its metrics argued over. Noor stepped back to the whiteboard and drew a loop. "Everything we have done fits somewhere on this circle," she said. "Projects fail when people treat it as a straight line that ends when the model is trained."

This lesson walks through the machine learning development lifecycle, the components of an ML pipeline, where models can come from, the ways a model can be used in production, and the AWS services that fit each stage. The exam guide expects you to recognize each stage and match services to it.

Stage 1: define the business goal

Every project starts with a business problem, not a technique. Andre Leclerc stated it in plain words: missed appointments leave slots empty and waste staff time; reduce them. A good goal names a measure of success and a constraint. Cascade Ridge's goal was to reduce the no-show rate by at least 1 appointment in every 100 within six months of the pilot, without refusing or penalizing any client. Agreeing on this first stops a team from building an impressive model that answers a question nobody asked.

Stage 2: frame the ML problem

Next, the business goal is translated into an ML task. What will the model predict, from what inputs, and how will the prediction be used? Here: predict, the evening before, the probability that each booking will be missed, using information known at booking time, so that front-desk staff can send an extra reminder or offer the slot to a waitlist. This is supervised binary classification. Framing also sets the evaluation metric, recall at a precision the front desk can handle, and the inference style, a nightly batch.

Framing is also where Noor asks whether ML is needed at all. If a simple rule would do, such as "call everyone booked more than three weeks ahead", it should be tested as the baseline. Lesson 11 develops this question.

Stages 3 and 4: collect and prepare data

Data is gathered from source systems, checked for quality and prepared, as lesson 3 described: cleaning, handling missing values, removing leaky fields, engineering features and splitting. Exploratory data analysis (EDA), looking at distributions, correlations and odd values with charts and summaries, happens here and often changes the framing. Jonah's EDA showed that misses cluster on Monday mornings and the first sunny Saturday of spring, which suggested two new features.

Stage 5: train and tune

The prepared training data is used to fit candidate models, and hyperparameters are tuned on validation data. Teams usually try several algorithms, starting with simple ones, and record every experiment: which data, which features, which settings, which result. Without that record, nobody can reproduce the best model a month later.

Stage 6: evaluate

The chosen model is evaluated on the test set with the agreed technical metrics and checked against the business goal and fairness concerns, as lessons 5 and 6 covered. Evaluation also includes a human review of examples: Dr. Kaur and Andre looked at 50 flagged bookings to see whether the reasons made sense.

Stage 7: deploy

Deployment makes the model available to the application that needs it, using one of the inference types from lesson 4. It also includes the surrounding work: the code that pulls tomorrow's bookings, the permissions the scoring job needs, the screen where the front desk sees the list, and a way to roll back.

Stage 8: monitor and retrain

After deployment, the model is watched for drift in its inputs and its accuracy, and for failures in the systems around it. When performance falls, new data is collected and the model is retrained, which closes the loop. Lesson 8 covers these practices, known as MLOps.

Where models come from

The exam guide asks you to recognize the sources of models, especially foundation models.

• Train a custom model on your own data, as Noor did for the no-show model. You control everything and need data, skills and computing time.
• Use a pre-trained open-source or open-weight model, downloaded from a model hub or through SageMaker JumpStart, and run it yourself, possibly after fine-tuning it on your data.
• Use a proprietary foundation model through an API, such as the models offered in Amazon Bedrock, without managing the model at all.

For traditional ML on Cascade Ridge's own tabular data, custom training is the norm, because no pre-trained model knows its appointment patterns. For language and vision tasks, starting from a pre-trained model is usually far cheaper than training from scratch.

Using a model in production: managed or self-hosted

There are two broad ways to serve a model to applications.

• A managed API service runs the model for you. You send a request and receive a result, paying per use. Amazon Bedrock for foundation models and services such as Amazon Comprehend or Amazon Transcribe for specific tasks work this way. There are no servers to manage, but you have less control over the model and its hardware.
• A self-hosted API means you deploy the model on infrastructure you manage, such as a SageMaker AI endpoint or containers on EC2 or Amazon EKS. You choose the model, the hardware and the scaling, and you carry the operational work and its cost, including paying for capacity even when it is idle.

The trade-off follows the AWS shared responsibility model from the Cloud Practitioner courses: the more AWS manages, the less control and the less work you have.

AWS services by stage

Two names in the table appear in the current exam guide and are newer than many study materials. Amazon Quick is AWS's AI-powered workspace for analysing data, building dashboards and working with AI agents over a company's connected data; it evolved from Amazon QuickSight, which continues inside it as Quick Sight for business intelligence. Kiro is AWS's AI-assisted development environment, in which coding agents help write, test and document software from specifications. Neither builds models for you, but both are places where the output of the lifecycle reaches people: Lorraine could see the pilot's results in a dashboard, and Kwame could use an assistant to write the scoring job's code. Product names in this area change often, so check the current service pages before an exam.

# A simple pipeline: each stage is a function, run in order and logged.
log = []
def stage(name):
    def wrap(fn):
        def run(x):
            out = fn(x); log.append(name); return out
        return run
    return wrap

@stage("collect")
def collect(_): return [(14, 1, 1), (2, 0, 0), (21, 2, 1), (3, 0, 0), (400, 0, 1)]
@stage("prepare")
def prepare(rows): return [r for r in rows if r[0] <= 365]
@stage("train")
def train(rows): return 10   # learned threshold: days ahead >= 10 means higher risk
@stage("evaluate")
def evaluate(t): return t, "recall check passed"

result = evaluate(train(prepare(collect(None))))
print("stages run:", " -> ".join(log))
print("model:", result)

stages run: collect -> prepare -> train -> evaluate
model: (10, 'recall check passed')

Real pipelines are built with tools such as SageMaker Pipelines, but the idea is the same: each stage is a defined step with inputs and outputs, run in order and recorded. Writing the lifecycle down as code is what makes it repeatable, which is the starting point for MLOps in the next lesson.

Who does what

The lifecycle involves more people than the data scientist. At Cascade Ridge, Andre owned the business goal and the definition of a no-show. Jonah, as the person closest to the booking data, owned data questions and EDA. Noor owned framing, training and evaluation. Kwame owned the scoring job, the integration with BookVet and the deployment. Hannah owned the privacy review and access, and Dr. Kaur held a veto over anything that touched clinical care. Mateo, from the cloud team, provided the AWS accounts, roles and monitoring. Larger organizations give these jobs names such as data engineer, ML engineer and product owner. The exam does not test job titles, but questions about who should approve a model, or who is responsible for monitoring it, assume that these roles exist and that each has an owner.

Noor wrote the owner of each stage on the whiteboard loop. When a stage has no owner, she said, it is the stage that quietly stops happening, and in most failed projects it is the last one, monitoring.

Documenting decisions along the way

Each stage produces decisions that someone will later need to understand: why this label definition, why these features, why this metric and threshold, why this inference type. Noor kept a short decision log beside the code, one line per decision with the date and the person who agreed it. When Hannah later asked why postal code was not used, the answer and its date were there in seconds. This habit feeds directly into model documentation such as SageMaker Model Cards, covered in lesson 9 and AWS-114, and it is the kind of record regulators and auditors increasingly expect for AI systems.

How long each stage takes

Newcomers often expect training to take most of a project's time. In Noor's experience, and in most published surveys of practitioners, the opposite is true. For the no-show model, defining the goal and the label took a week of conversations, finding and cleaning the data took about three weeks, training and tuning took a few days, evaluation and review with staff took a week, and building the nightly job, the front-desk list and the monitoring will take most of December and January. The model itself was the quickest part. Planning a project with that shape in mind, and telling stakeholders early, prevents the common disappointment of a model that is "finished" in October but not usable until February.