Domain 3 review: customization and evaluation

The second half of domain 3 covers how foundation models are trained and adapted, and how to tell whether a model or an application built on one is doing its job. These two task statements, 3.3 and 3.4, generate many of the exam's trickier questions, because the options are often all real techniques and the difference lies in cost, data needs or what each one actually measures. In AWS-113 Jonah watched Noor evaluate three candidate models for the after-hours summaries against a rubric the emergency hospital's technicians helped write. That evaluation is the running example for this lesson.

Task 3.3: the training landscape

Pre-training is the first, enormous stage of building a foundation model: training on very large amounts of general, mostly unlabelled data so the model learns language, images or code broadly. Fine-tuning takes a pre-trained model and trains it further on a smaller, usually labelled data set so it performs better at a particular task or in a particular style. Continued pre-training, which the guide also calls continuous pre-training, trains a pre-trained model further on unlabelled text from a specific domain, such as veterinary literature, so it learns that domain's vocabulary and knowledge. Distillation transfers capability from a large teacher model to a smaller student model, so the smaller one can do a specific task more cheaply.

The guide names several methods of fine-tuning. Instruction tuning trains a model on examples of instructions paired with good responses, so it follows instructions better. Domain adaptation adjusts a model to a specialized field. Transfer learning is the general idea behind all of these: reusing what a model learned on one task as the starting point for another. Reinforcement learning from human feedback, or RLHF, uses people's ratings of model outputs to train the model toward responses people prefer, and it is one of the ways general assistants are made more helpful and less harmful.

Preparing data for fine-tuning

The guide lists what matters when preparing data: curation, governance, size, labelling, representativeness and RLHF. Curation means selecting and cleaning examples carefully, because a fine-tuned model learns whatever patterns the examples contain, including mistakes. Governance means knowing where the data came from, whether you have the right to use it and whether it contains personal information. Size matters, but quality matters more: a few hundred excellent examples can do more than thousands of inconsistent ones. Labels must be accurate and consistent. Representativeness means the examples cover the real range of cases the model will see. A summary model fine-tuned only on calls about dogs, for example, may handle a call about a rabbit or a horse badly.

Noor and Jonah discussed whether the group should fine-tune a model for the after-hours summaries. They decided not to, at least for now. Few-shot prompts in a template gave summaries in the right structure, and the emergency hospital's notes contain client names and phone numbers that would need careful removal before they could become training data. That is a typical exam pattern too: fine-tuning is rarely the first answer, and when it appears as a distractor, the scenario usually has a cheaper fix.

Task 3.4: evaluating model performance

The guide names three approaches to evaluating foundation models. Human-in-the-loop evaluation has people rate outputs for quality, accuracy, tone and safety; it is slower and more expensive but captures judgement that automated metrics miss. Benchmark data sets are standard collections of questions or tasks with known answers, used to compare models on the same footing. Amazon Bedrock Model Evaluation runs evaluations for you, either automatically, with metrics and built-in or custom data sets, or with human reviewers, including your own staff, and it supports using a model as a judge.

The exam expects you to know what the named metrics measure. ROUGE, which stands for Recall-Oriented Understudy for Gisting Evaluation, measures how much of a reference text's wording appears in the generated text, and it is most associated with summarization. BLEU, the Bilingual Evaluation Understudy, measures how closely generated text matches reference translations, and it is most associated with machine translation. BERTScore compares the meaning of generated and reference text using embeddings, so it can reward a correct paraphrase that shares few exact words. LLM-as-a-judge uses a strong model to grade outputs against a rubric, which scales better than human review but must itself be checked.

# A toy word-overlap score in the spirit of ROUGE-1 recall (illustration only)
reference = "dog vomited twice after eating chocolate owner advised to come in now"
generated = "owner advised to bring the dog in now after it vomited twice from chocolate"
ref, gen = reference.split(), set(generated.split())
overlap = sum(1 for w in ref if w in gen)
print("reference words found:", overlap, "of", len(ref), "->", round(overlap / len(ref), 2))

reference words found: 10 of 12 -> 0.83

The toy score above shows both the value and the limit of overlap metrics. It rewards a summary that keeps the important words of the reference, but it would also reward a summary that kept the words while reversing the meaning. That is why Noor's evaluation of the three candidate models combined an automatic score with a rubric scored by two technicians: did the summary include every symptom mentioned, did it add anything not in the notes, and was the owner's callback number correct.

Evaluating applications, not just models

The current guide also asks you to evaluate applications built with foundation models, such as RAG systems, agents and workflows, and to align them with business objectives. A RAG system can fail in two places: retrieval can bring back the wrong passages, or generation can misuse the right ones. So you evaluate both whether the retrieved passages are relevant and whether the answer is faithful to them. An agent is evaluated on whether it completes the task, whether it chooses the right tools, and whether it stays within its permissions. Business alignment metrics named in the guide include task completion rate, user satisfaction and cost per interaction, alongside broader objectives such as productivity and user engagement.

Fine-tuning on Amazon Bedrock and SageMaker AI

Both Bedrock and SageMaker AI support customization, and the exam may ask which to use. Amazon Bedrock offers fine-tuning and continued pre-training for supported models as a managed feature: you provide training data in Amazon S3, Bedrock creates a private copy of the model customized for you, and you run it with provisioned throughput. Your data is not used to improve the base model for others. Bedrock also supports model distillation for supported models. SageMaker AI, including JumpStart, gives more control over the training process and supports a wider range of open models and custom training code, at the cost of more operational work. The usual exam pattern: if the scenario wants a managed customization with minimal infrastructure, Bedrock; if it wants full control over training or a model Bedrock does not offer, SageMaker AI.

A common trap is to assume that customization is the default way to make a model fit an organization. The guide's own ordering of costs tells the story: in-context learning and RAG are cheap and quick to change, while fine-tuning needs curated data, training runs, provisioned capacity and a plan to repeat the work when the base model is updated. Choose customization when prompting and retrieval have been tried and a measurable gap remains, such as a consistent house style that examples alone cannot achieve.

Evaluating RAG systems and agents in practice

For a RAG system, Noor uses a small test set of real questions with known correct passages and answers. She measures how often the right passage appears among the first few retrieved, which tests retrieval, and whether the answer stays faithful to the passages, which tests generation. Bedrock evaluation features include options for evaluating retrieval and generation in knowledge bases, and LLM-as-a-judge evaluators can score faithfulness and relevance at scale, with human spot checks. For agents, she looks at task completion, the number of steps taken, whether the agent called the right tools with valid inputs, and whether it ever attempted an action outside its permissions. Each of these maps to a business question: does it work, is it efficient, and is it safe?

Benchmark data sets have a place too. Public benchmarks let you compare candidate models on general skills such as reasoning or reading comprehension before you test them on your own data. But a model that scores well on a public benchmark may still do poorly on veterinary notes full of abbreviations. That is why the guide pairs benchmarks with human evaluation and business metrics rather than treating any one of them as enough.

When a question asks how to prepare data for fine-tuning, think of the data as a teacher. It must be accurate, because the model will copy its errors; consistent, because mixed styles produce mixed results; representative, because gaps become blind spots; and lawful, because data you had no right to use creates legal risk. Removing personal information, recording where every example came from and keeping a held-out set for evaluation are all signs of good governance that exam answers often mention.

A last point on evaluation: decide what good looks like before you test. Noor and the technicians agreed the rubric, the pass mark and the test questions before any model was tried, so that no one could adjust the standard to suit a favourite model afterwards. The exam rewards the same discipline: define the objective and the metric first, then choose the model or approach that meets it.

Practice items for the second half of domain 3

First: a company wants a model to use the specialized vocabulary of marine insurance, and it has millions of pages of unlabelled policy documents but few labelled examples. Which method fits? Continued pre-training on the domain text. Second: a team needs to compare translations from two models into French against professional reference translations. Which metric is designed for this? BLEU. Third: a startup wants the capability of a large model on one narrow task but at a fraction of the inference cost. Which approach fits? Distillation. Fourth: a support assistant's answers score well on BERTScore but customers say it does not solve their problems. Which metric would reveal this? A business alignment metric such as task completion rate or user satisfaction. Fifth: what does RLHF use to train the model? Human ratings of model outputs.