Techniques that improve model output

The first test drafts for the product description pilot are disappointing in an instructive way. Jules pastes a supplier spec sheet for a new rain shell into a general chat model and asks for a description. The result reads well, but it calls the jacket "fully waterproof with taped seams" when the spec sheet says water-resistant with a durable water repellent finish, and it mentions a lifetime warranty that Birchline does not offer. The model filled gaps with likely-sounding details. "That is not the model being broken," Farah says. "That is the model doing what it was built to do, which is produce plausible text. Our job is to give it what it needs to produce accurate text, and to check it." This lesson covers the techniques leaders need to understand, so they can ask the right questions of technical teams.

Why model output goes wrong

Generative models learn patterns from very large amounts of data and then predict likely continuations. That gives them broad knowledge and fluent language, but also built-in limitations that every leader should be able to name.

None of these disappears completely. The techniques below reduce them, and the human review in lesson 4 catches what remains. A useful way to think about the techniques is as a ladder from cheapest and fastest to most involved.

Rung one: prompt engineering

Prompt engineering is designing the instructions and context given to a model so that it produces the intended output. It costs nothing but thought, can be changed in minutes, and often produces the largest single improvement. Leaders do not need to write prompts themselves, but they should recognize the main techniques by name.

• Zero-shot prompting gives an instruction with no examples: "Write a product description for this jacket."
• One-shot and few-shot prompting include one or several examples of good output, so the model can copy the format, length and tone. Birchline adds three approved past descriptions to every prompt.
• Role prompting tells the model who it is acting as: "You are a copywriter for an outdoor retailer whose tone is practical and friendly."
• Prompt chaining splits a task into steps, with the output of one step feeding the next: first extract the facts from the spec sheet, then write the description from only those facts, then check the description against the facts.
• Chain-of-thought prompting asks the model to work through intermediate reasoning before answering, which tends to help on multi-step problems.
• ReAct (reason and act) combines reasoning with actions such as looking something up or calling a tool, then reasoning about the result. It underlies many agents.

The improved Birchline prompt uses several of these together. It sets the role, states the constraint "use only the facts provided; if a fact is missing, write MISSING rather than guessing", gives three examples, and chains extraction before writing. The invented warranty disappears, because the prompt removes the model's room to fill gaps, and because the extraction step produces a list of facts that the editor can check quickly.

Sampling settings

Alongside the prompt, most models accept settings that change how they choose words. Leaders meet these in product consoles and vendor discussions, so knowing what each does is useful.

For product descriptions, Birchline uses a low temperature so that two runs produce similar facts, and a maximum length that fits the 160-word limit with a little room. For the marketing team's slogan brainstorming, a higher temperature is fine, because variety is the point and a person picks the best idea.

Rung two: grounding and retrieval-augmented generation

Grounding means connecting a model's output to verifiable sources of information, so that it answers from facts rather than from memory alone. Grounding can use first-party data, which is the organization's own information such as Birchline's catalogue and policies; third-party data licensed from others, such as a supplier's specification database; or world data, such as current results from web search.

The most common grounding approach is retrieval-augmented generation (RAG). When a question arrives, a retrieval step searches a collection of documents for the passages most relevant to it. Those passages are added to the prompt, and the model is instructed to answer from them and to cite them. Because the documents can be updated at any time, RAG also works around the knowledge cutoff: the store policy assistant can answer about last month's policy change as soon as the new policy is in the collection.

Google Cloud offers grounding at several levels, which GCP-111 describes in more detail: prebuilt retrieval through its enterprise search offering, RAG building blocks for developers on its AI platform, and grounding with Google Search for current public information. The leadership point is simpler. Grounding is usually the right answer when the problem is "the model does not know our facts" or "the model's facts are out of date", and it has a quality requirement of its own: the documents must be accurate, current and properly permissioned, which is lesson 5's subject.

Grounding also makes review faster. When the policy assistant cites the paragraph its answer came from, a store manager can check it in seconds. Birchline makes citation a requirement for every discovery use case.

# A toy retrieval step: score each policy passage by shared words with the question.
passages = {
    "returns-1": "Unworn items can be returned within 60 days with a receipt.",
    "returns-2": "Worn footwear can be returned within 30 days if faulty.",
    "giftcards": "Gift cards cannot be exchanged for cash except where law requires.",
    "pricing":   "Price adjustments are given within 14 days of purchase.",
}
question = "Can a customer return worn boots that are faulty?"
q = set(question.lower().replace("?", "").split())
def overlap(text):
    return len(q & set(text.lower().replace(".", "").split()))
ranked = sorted(passages.items(), key=lambda kv: -overlap(kv[1]))
for pid, text in ranked[:2]:
    print(pid, overlap(text), "|", text)

returns-2 3 | Worn footwear can be returned within 30 days if faulty.
returns-1 2 | Unworn items can be returned within 60 days with a receipt.

Real systems use far better retrieval than counting shared words, typically embeddings, numeric representations of meaning that let the system find passages about "faulty footwear" even when the question says "broken boots". The idea is the same: find the best evidence first, then ask the model to answer from it.

Rung three: fine-tuning

Fine-tuning continues training an existing model on a set of examples so that it learns a specialized style, format or task. Where prompting tells a model what to do each time, fine-tuning changes the model's tendencies. It can help when the organization has many good examples of a consistent task, when prompts have become very long, or when a smaller tuned model can match a larger general one at lower cost. Related lighter-weight methods, sometimes called parameter-efficient tuning, adjust only a small part of the model and need fewer resources.

Fine-tuning has costs leaders should weigh. It needs a large, clean, representative set of examples, often hundreds or thousands. It takes time and expertise to evaluate. It must be repeated when the base model is upgraded. And it is the wrong tool for teaching a model facts that change: a model tuned on this year's product line will be wrong next year, while a grounded model simply retrieves the new catalogue. Farah's summary for Jules is a rule of thumb: ground for knowledge, tune for behaviour, and try prompting first for both.

Rung four is not a rung: review and monitoring

The side rail of Farah's ladder runs the whole height: human-in-the-loop review and continuous monitoring. Every technique reduces errors; none removes them. Lesson 4 covers review workflows. Monitoring, covered in lesson 11, watches quality over time, because model providers upgrade models, documents change, and the kinds of questions users ask drift away from what was tested. Version control for prompts and settings, tracked KPIs, and a regular sample of outputs checked by people keep a system good after launch.

With the improved prompt, grounding in the catalogue and three examples, the second test batch of 50 drafts has no invented warranties and only two factual errors, both caused by a supplier spec sheet that was itself wrong. That last finding goes to the data readiness work in lesson 5. Fine-tuning is not needed for now; Farah notes it as an option to revisit if Birchline ever wants a smaller, cheaper model for very high volumes.

Asking good questions of technical teams

Leaders rarely tune prompts or build retrieval themselves, but they approve the work and the budget. Farah gives Jules a short set of questions that reveal whether a team has chosen sensible techniques. What specific errors did you see before this change, and how often? Which technique did you try first, and what did it fix? How are the grounding documents kept current, and who owns them? What happens when the model has no good source for an answer? How do you know the change did not make something else worse? A team that can answer these clearly has usually climbed the ladder in the right order. A team that jumps straight to "we need to fine-tune a model" without describing the errors it is fixing probably has not.