Customizing models: fine-tuning and continued pre-training

A vendor demonstration had left Lorraine Beaudry, the CFO, with a question she brought to the eighth AI hour. "The salesperson said we should train our own model on all our records so it really understands veterinary work," she said. "Is that what we should be doing?"

Noor drew a ladder on the whiteboard. On the bottom rung she wrote PROMPT. Then RAG, FINE-TUNE, CONTINUED PRE-TRAINING and, at the top, PRE-TRAIN FROM SCRATCH. "Each rung costs more, takes longer and needs more data and skill than the one below," she said. "We climb only when the rung below has been tried and measured and is not good enough. So far, we have not needed to leave the second rung."

This lesson explains the ways a foundation model can be customized, what each changes, how to prepare data for them, and the cost trade-offs the exam guide asks about. It also explains why most business applications, including both of Cascade Ridge's, never need to climb very high.

What each approach changes

The approaches differ in one fundamental way: whether they change the model's weights, the billions of numbers learned in training.

• In-context learning (prompt engineering, including few-shot examples) changes nothing in the model. All the guidance lives in the prompt. It is quick, cheap to try and easy to change, but every request pays for the extra tokens.
• Retrieval-augmented generation also changes nothing in the model. It adds retrieved knowledge to the prompt, which suits facts that change and must be cited.
• Fine-tuning continues training an existing foundation model on a smaller, labelled dataset of examples, changing its weights so it gets better at a specific task, style or format.
• Continued pre-training (sometimes called continuous pre-training) continues the model's original kind of training on a large amount of unlabelled text from a particular domain, so it absorbs that domain's vocabulary and patterns.
• Pre-training from scratch builds a new foundation model from random weights on an enormous dataset. It costs the most by far and is done mainly by model providers and very large organizations.
• Distillation trains a smaller "student" model to imitate the outputs of a larger "teacher" model on a task, producing a model that is cheaper and faster to run while keeping much of the teacher's quality for that task.

Fine-tuning

Fine-tuning is the customization most teams consider first once prompting is not enough. The exam guide names several forms of it.

• Instruction tuning fine-tunes a model on examples of instructions paired with good responses, so it follows a particular kind of instruction better. Many base models are instruction-tuned by their providers before release; a business might instruction-tune further for its own tasks.
• Domain adaptation fine-tunes a model on examples from a specialized field, such as veterinary medicine or law, so it handles that field's terms and conventions better.
• Transfer learning is the general idea behind all of these: knowledge learned on one broad task is transferred and adapted to a new, narrower one, so far less data is needed than training from scratch.

A fine-tuning dataset is usually a set of prompt and completion pairs, often stored as JSON Lines, one example per line. For a hypothetical Night Notes fine-tune, each example would pair an invented call note with the summary a veterinarian had approved.

import json

examples = [
    {"prompt": "Lab, 4 yrs, 30 kg. Ate 1 pk sugar-free gum ~30 min ago. Drooling, wobbly. Owner Priya.",
     "completion": "Patient: dog, Labrador, 4 years, 30 kg | Concern: ate one packet of sugar-free gum; now drooling and unsteady | Timing: about 30 minutes before the call | Actions taken by owner: none mentioned | Advice given: unclear"},
    {"prompt": "Cat 12y, not eating 2 days, hiding. No toxins known. Owner Marc.",
     "completion": "Patient: cat, 12 years | Concern: not eating for two days and hiding; no toxins known | Timing: started two days before the call | Actions taken by owner: none mentioned | Advice given: unclear"},
]
lines = [json.dumps(e) for e in examples]
print("\n".join(lines))
words = sum(len((e["prompt"] + " " + e["completion"]).split()) for e in examples)
print("Examples:", len(examples), " Total words:", words)

{"prompt": "Lab, 4 yrs, 30 kg. Ate 1 pk sugar-free gum ~30 min ago. Drooling, wobbly. Owner Priya.", "completion": "Patient: dog, Labrador, 4 years, 30 kg | Concern: ate one packet of sugar-free gum; now drooling and unsteady | Timing: about 30 minutes before the call | Actions taken by owner: none mentioned | Advice given: unclear"}
{"prompt": "Cat 12y, not eating 2 days, hiding. No toxins known. Owner Marc.", "completion": "Patient: cat, 12 years | Concern: not eating for two days and hiding; no toxins known | Timing: started two days before the call | Actions taken by owner: none mentioned | Advice given: unclear"}
Examples: 2  Total words: 102

On AWS, fine-tuning can be done in Amazon Bedrock for models that support it, by pointing a customization job at training and validation data in Amazon S3, or in Amazon SageMaker AI, which gives full control over the training process and hosting. A customized model on Bedrock is a separate model resource that the application calls instead of the base model, and depending on the model it may need purchased capacity, such as Provisioned Throughput, to serve requests. That ongoing hosting cost is easy to overlook when comparing options.

Continued pre-training

Continued pre-training uses unlabelled text: no prompts and answers, just a large volume of documents from a domain. A provider of veterinary software might continue pre-training a model on many thousands of textbooks, journal articles and anonymized records so the model learns the field's language deeply. The cost is high because the volume is large, and the result still usually needs instruction tuning or good prompting to follow instructions well.

For Cascade Ridge, continued pre-training made no sense. Its own text, 140 SOPs and some years of notes, was far too small to shift a large model's knowledge, and much of it contained personal information that Hannah would not allow into training without a full privacy review and de-identification.

Preparing data for fine-tuning

The exam guide lists the qualities that matter when preparing data to fine-tune a model. Each one is a place where projects go wrong.

• Curation. Choose examples that are correct and representative of the task, and remove duplicates, errors and examples that teach the wrong behaviour. A few hundred excellent examples can beat thousands of careless ones.
• Governance. Know where every example came from, whether you have the right to use it, and whether it contains personal or confidential information. Record who approved the dataset, and protect it with the same access controls as the source data.
• Size. Enough examples to cover the variety of real inputs. The right number depends on the task and model; providers publish guidance, and teams usually start small and measure.
• Labelling. Each example needs the right output, usually written or checked by experts. For Night Notes, that would mean veterinarians approving each summary, which is slow and expensive.
• Representativeness. The examples should reflect the real mix of inputs: cats and dogs, short and long notes, typos, different clinics. A dataset of only easy cases produces a model that fails on hard ones. Representativeness also matters for fairness, which AWS-114 covers.

Reinforcement learning from human feedback (RLHF) is a related technique used mostly by model providers. People compare or rate model responses, those preferences train a reward model, and the foundation model is then tuned to produce responses people prefer, such as ones that are more helpful and less harmful. It is one reason modern models follow instructions and decline harmful requests as well as they do.

Cost trade-offs

The exam guide asks you to explain the cost trade-offs between pre-training, fine-tuning, in-context learning, RAG and distillation. Lorraine wanted the comparison in numbers, so Jonah built a rough model for Night Notes at 3,000 summaries a month. Every figure below is illustrative, chosen to show the shape of the trade-off rather than real prices.

# Illustrative monthly cost comparison for 3,000 Night Notes summaries (all figures invented).
requests = 3000
options = {
    # name: (one-time setup cost, monthly fixed cost, cost per request)
    "Prompt with 1 example":      (0,     0,    0.0040),
    "Prompt with 5 examples":     (0,     0,    0.0075),
    "Fine-tuned model":           (2500,  1800, 0.0030),
    "Distilled smaller model":    (1200,  900,  0.0012),
}
for name, (setup, fixed, per) in options.items():
    monthly = fixed + per * requests
    year = setup + 12 * monthly
    print(f"{name:26} monthly {monthly:8.2f}  first year {year:9.2f}  (illustrative USD)")

Prompt with 1 example      monthly    12.00  first year    144.00  (illustrative USD)
Prompt with 5 examples     monthly    22.50  first year    270.00  (illustrative USD)
Fine-tuned model           monthly  1809.00  first year  24208.00  (illustrative USD)
Distilled smaller model    monthly   903.60  first year  12043.20  (illustrative USD)

At this volume, the simple prompt was cheapest by a wide margin. The fine-tuned model's per-request cost was lower, but its fixed hosting cost swamped the saving at 3,000 requests a month. The picture would change at much higher volumes, which is exactly why the trade-off depends on the workload. Noor summarized the pattern on the whiteboard:

Fine-tuning and RAG solve different problems, and confusing them is the most common mistake. Fine-tuning teaches behaviour: a format, a tone, a way of handling a task. RAG supplies knowledge: facts that change and must be cited. A fine-tuned model does not reliably memorize facts, cannot cite them and must be retrained when they change. Many strong applications combine both: a model fine-tuned for format, answering from retrieved passages.

Importing and hosting customized models

Not every customized model is trained on Bedrock. Some teams fine-tune an open-weight model in SageMaker AI or elsewhere and then bring it into Bedrock, where a model import feature lets supported model types be served through the same API as other Bedrock models. Others keep the customized model on a SageMaker AI endpoint. Either way, the team becomes responsible for evaluating the customized model as carefully as any new model, and for repeating that evaluation whenever it is retrained. A customized model is a new model, with its own strengths and failure modes.

Lorraine wrote one word on her notepad, LADDER, and underlined it. "I'll use that the next time a vendor calls," she said.