The generative AI project lifecycle

Lorraine Beaudry, the CFO, had one question when the board asked for an update on the overnight call summaries. "Where are we in the project, and how will we know when it is finished?" Jonah realized he could describe what the tests had found but not where they fitted in a plan. Noor said that was the right moment to look at the lifecycle.

The AWS Certified AI Practitioner exam guide describes the foundation model lifecycle as a series of stages: data selection, model selection, pre-training, fine-tuning, evaluation, deployment and feedback. Not every organization performs every stage; a veterinary group will never pre-train a model. But knowing the full cycle shows which stages are someone else's work, which are yours, and why a generative AI project is never really finished. This lesson walks through the stages, then maps them onto a practical project plan for Cascade Ridge, and introduces context engineering, the work of deciding exactly what information a model sees with each request.

Data selection

Every model is shaped by its data. For a model provider, data selection means assembling, filtering and cleaning the enormous collection used for pre-training: removing duplicates, low-quality pages, private information and harmful content, and deciding the mix of languages, code and subjects. The choices here determine much of what the model knows and many of its biases.

For a business adapting a model, data selection happens on a much smaller scale but matters just as much. Cascade Ridge's relevant data includes the call notes used for testing, the procedure manual for the question tool, and any examples gathered for fine-tuning later. Hannah Weiss set the rules for this data before anyone touched it: notes used for testing are de-identified, owner names and phone numbers are removed, and nothing leaves the group's AWS accounts.

Model selection

Model selection means choosing an existing foundation model to build on. This is where most organizations enter the lifecycle. The choice depends on the task's modality, the quality needed, cost, latency, the context window, the languages supported, availability in the required Region, and the provider's terms on data use. Lesson 10 covers these trade-offs. In practice, teams shortlist two or three candidates and compare them on their own test examples rather than trusting public leaderboards, which measure general ability, not ability on veterinary call notes.

Pre-training

Pre-training is the expensive first training stage, in which a model learns from its huge, broad dataset to predict the next token. It needs thousands of specialized processors running for weeks or months, and it is done by a small number of model providers. For an organization like Cascade Ridge, pre-training is part of the model's history, not part of the project. Some large organizations do continued pre-training, feeding a model a large amount of unlabelled domain text, such as millions of veterinary records, to deepen its domain knowledge; this is rare and costly.

Fine-tuning

Fine-tuning continues training a pre-trained model on a smaller, task-specific dataset, usually examples of inputs paired with the desired outputs. It changes the model's parameters so that it performs one kind of task in a particular way: writing summaries in the group's house style, for example, or using its abbreviations correctly. Providers also fine-tune their own models to follow instructions and to behave safely, a stage often called alignment.

Fine-tuning is powerful but not the first tool to reach for. It needs a few hundred to several thousand good examples, costs money for training and for hosting the custom model, and must be repeated when the base model changes. Most needs can be met more cheaply by better prompts and by grounding with retrieval. Noor's ordering became Jonah's rule: try prompting first, add retrieval if the model needs facts it does not have, and consider fine-tuning only if a consistent style or behaviour still cannot be achieved.

Evaluation

Evaluation measures whether the model does the job well enough. For a provider, it means benchmarks across many tasks and safety testing. For a business, it means testing on realistic examples of its own task. Evaluation takes three common forms, and good projects use more than one.

• Human evaluation. Qualified people score outputs, as Dr. Kaur did with her red pen. It is the most trusted and the slowest.
• Automated metrics. Scores computed by code, such as whether required facts are present, or text-similarity measures such as ROUGE for summaries. They are fast and repeatable but measure only what they are designed to measure.
• Model-based evaluation. A second model, sometimes called an LLM-as-a-judge, scores outputs against criteria. It scales well but inherits model weaknesses, so its scores are checked against human ones.

Amazon Bedrock includes model evaluation features that run automatic, model-based and human evaluation jobs against a dataset you supply, so teams can compare candidate models and prompts on the same examples.

# A tiny evaluation harness: score candidate outputs on required facts.
tests = [
    {"note": "raisins, no vomiting, en route", "required": ["raisin", "vomit", "en route"]},
    {"note": "may have eaten sock, antacid given", "required": ["may have", "sock", "antacid"]},
    {"note": "cat not eating 2 days, hiding", "required": ["not eating", "2 days", "hiding"]},
]
candidates = {
    "Model A": ["Beagle ate raisins; not vomiting; owner en route.",
                "Dog may have eaten a sock; owner gave antacid.",
                "Cat not eating for 2 days and hiding."],
    "Model B": ["Beagle ate raisins; owner en route.",
                "Dog ate a sock.",
                "Cat not eating for 2 days; hiding under bed."],
}
for name, outs in candidates.items():
    hits = sum(all(r in o.lower() for r in t["required"]) for t, o in zip(tests, outs))
    print(f"{name}: {hits} of {len(tests)} summaries contain every required fact")

Model A: 3 of 3 summaries contain every required fact
Model B: 1 of 3 summaries contain every required fact

Deployment

Deployment puts the model into use. With a managed service such as Amazon Bedrock, deployment mostly means calling the model from an application through an API, with permissions, logging and guardrails in place. With self-hosted models in SageMaker AI, it also means choosing instances, setting up endpoints and scaling them. Deployment choices include whether requests are handled in real time or in batches; overnight call summaries could be produced in a batch at the end of each shift, which is often cheaper than real-time processing, as lesson 10 explains.

For Cascade Ridge, deployment also meant people and process: who reviews summaries, where they appear in the morning handover, and what happens when one looks wrong. Andre Leclerc, the operations manager, insisted that the pilot start in the emergency hospital only, with a written fallback: if the summaries are unavailable, the team reads the full notes as before.

Feedback and monitoring

The final stage loops back to the first. Feedback from users and monitoring of the live system show where the model falls short: summaries reviewers keep correcting, questions the tool answers poorly, costs creeping up. That feedback becomes new test examples, prompt changes, new documents for retrieval, or, eventually, fine-tuning data. Model providers also release new versions, so a project should plan to re-run its evaluation when it switches versions. This is why the lifecycle is drawn as a circle.

Jonah set up a simple feedback route: each summary in the pilot gets a thumbs-up or a correction box, and corrections are reviewed weekly with Noor. Lorraine had her answer: the project is at evaluation, deployment to a pilot is planned, and it will never be finished in the sense of being left alone; it will be maintained like any other clinical system.

Context engineering

The exam guide asks you to describe the role of context engineering in foundation model applications. Context engineering is the practice of deciding what goes into the model's context window for each request: the instructions, examples, retrieved documents, conversation history, tool results and any user details, in what order and at what length. Prompt engineering is part of it; context engineering is the wider job of assembling everything the model sees.

Good context engineering improves quality and cuts cost at the same time. For the overnight summaries, the context is the fixed instructions, two short example summaries in the house style, and the call note itself, nothing else. For the procedure manual tool, it is the instructions, the three most relevant chunks rather than the whole manual, and the manager's question. Including irrelevant material wastes tokens and can distract the model; leaving out a needed fact invites a guess.

# Assembling a context within a token budget (rough count: 4 characters per token).
budget = 600
parts = [
    ("instructions", "Summarize the call note using only its facts. Keep hedges. " * 3),
    ("example summary", "Patient: ... Concern: ... Timing: ... Actions: ... Advice: ... " * 4),
    ("call note", "Owner says beagle may have eaten raisins around 9 pm; no vomiting. " * 6),
    ("older history", "Previous visits and vaccinations listed in full. " * 40),
]
used, chosen = 0, []
for name, text in parts:
    cost = len(text) // 4
    if used + cost <= budget:
        used += cost
        chosen.append(name)
print("included:", chosen)
print("estimated tokens used:", used, "of", budget)

included: ['instructions', 'example summary', 'call note']
estimated tokens used: 207 of 600

The budget left out the older history, which was the right call for a summary of one night's phone call. If Dr. Kaur later wants summaries to mention relevant history, the team will retrieve only the related entries rather than the whole record.

Where Cascade Ridge sits in the lifecycle

Jonah drew the lifecycle circle on the whiteboard again and shaded the stages Cascade Ridge actually performs. Data selection, in the narrow sense of choosing and de-identifying test notes and documents: yes. Model selection: yes, from the models available in the group's Region. Pre-training: no, that belongs to the providers. Fine-tuning: not yet, and only if prompting and retrieval fall short. Evaluation, deployment and feedback: yes, and these are where most of the group's effort will go for as long as the tool is in use.

The shading made a point that surprised Lorraine. Most of the work in a generative AI project for a business like Cascade Ridge is not in the model at all. It is in preparing data, testing carefully, fitting the tool into people's work, and listening to feedback. Budgets and timelines that assume the hard part is "getting the AI" usually underestimate the project. Noor added that this is also why the group's machine learning experience with the no-show model transferred so well: the habits of defining success, measuring against a test set and monitoring in production are the same.