Prompting, retrieval and tool use

A trained model is only as useful as the way it is used. The same model can give a vague, wrong answer or a precise, well-sourced one depending on the instructions it receives, the information placed in its context, and whether it can call tools to look things up, calculate or take actions. This lesson covers how generation works at inference time, how to prompt models effectively, how retrieval-augmented generation grounds answers in documents, how tool use and agents extend what models can do, and the risks (hallucination, prompt injection, cost) that system designers must manage. It ends with the regulatory landscape for deploying generative AI.

How generation works

At inference time, the model processes the prompt (the prefill phase, which can run in parallel across all prompt tokens) and then generates the response one token at a time (the decode phase). At each step it computes a probability distribution over the vocabulary and a decoding strategy chooses the next token:

• Greedy decoding picks the most likely token; deterministic but often repetitive.
• Temperature sampling divides the logits by a temperature \(T\) before the softmax: \(T < 1\) sharpens the distribution (more predictable), \(T > 1\) flattens it (more varied).
• Top-k sampling keeps only the \(k\) most likely tokens; top-p (nucleus) sampling keeps the smallest set whose probabilities sum to at least \(p\).

import numpy as np

tokens = ["Paris", "Lyon", "the", "France", "a", "Marseille", "banana"]
logits = np.array([6.0, 3.5, 3.0, 2.5, 2.0, 1.5, -1.0])


def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()


def top_p(p, threshold):
    order = np.argsort(p)[::-1]
    keep = order[: int(np.searchsorted(np.cumsum(p[order]), threshold)) + 1]
    q = np.zeros_like(p)
    q[keep] = p[keep]
    return q / q.sum()


for T in [0.5, 1.0, 1.5]:
    p = softmax(logits / T)
    entropy = -np.sum(p * np.log2(p))
    print(f"T={T}: " + "  ".join(f"{t}={x:.2f}" for t, x in zip(tokens, p) if x >= 0.01) + f"   entropy {entropy:.2f} bits")

p = softmax(logits / 1.5)
for th in [0.9, 0.5]:
    q = top_p(p, th)
    kept = [t for t, x in zip(tokens, q) if x > 0]
    print(f"top-p {th} at T=1.5 keeps {kept}")
rng = np.random.default_rng(0)
draws = rng.choice(len(tokens), size=1000, p=p)
print("1000 samples at T=1.5:", {tokens[i]: int(np.sum(draws == i)) for i in range(len(tokens))})

T=0.5: Paris=0.99   entropy 0.10 bits
T=1.0: Paris=0.84  Lyon=0.07  the=0.04  France=0.03  a=0.02   entropy 0.97 bits
T=1.5: Paris=0.65  Lyon=0.12  the=0.09  France=0.06  a=0.04  Marseille=0.03   entropy 1.74 bits
top-p 0.9 at T=1.5 keeps ['Paris', 'Lyon', 'the', 'France']
top-p 0.5 at T=1.5 keeps ['Paris']
1000 samples at T=1.5: {'Paris': 622, 'Lyon': 131, 'the': 100, 'France': 67, 'a': 40, 'Marseille': 33, 'banana': 7}

Low temperatures suit factual answers, extraction and code; higher temperatures suit brainstorming and creative writing. Even at low temperature, outputs can vary between runs because of batching and floating-point effects in serving systems.

Generation speed is limited by memory bandwidth rather than arithmetic, because every decode step reads all the model weights and the key-value (KV) cache, which stores the attention keys and values of all previous tokens so they need not be recomputed. The KV cache grows with context length and can exceed the size of the model itself for long contexts.

Prompting

Prompting means writing the input so that the model produces what you need. Modern assistant models follow plain instructions well, so the most effective techniques are mostly about clear communication:

• Be explicit about the task, audience, format and length. "Summarize this contract for a small-business owner in five bullet points, flagging any automatic renewal clauses" works far better than "summarize this".
• Provide context: background, constraints, definitions and the relevant documents.
• Give examples (few-shot prompting) of inputs and desired outputs when the format or style is subtle. Models can learn a task from a handful of examples in the prompt, an ability called in-context learning.
• Structure long prompts with clear sections or tags separating instructions, documents and examples.
• Ask for reasoning when the task needs it: requesting step-by-step thinking (chain-of-thought prompting) improves accuracy on multi-step problems for many models, while reasoning models do this automatically and may only need a budget or effort setting.
• Specify what to do when information is missing, such as "if the document does not say, answer that it is not stated", which reduces fabricated answers.
• Iterate: test prompts on varied inputs, examine failures and refine.

System prompts set persistent instructions and persona for an application. Prompts are also an attack surface: users may try to override instructions, so applications should not rely on a prompt alone to enforce security.

Hallucination and grounding

LLMs sometimes produce fluent statements that are false, such as invented citations, wrong numbers or plausible but non-existent facts. This is often called hallucination (or confabulation). It arises because the model generates likely-sounding text from patterns rather than retrieving verified facts, and because training has historically rewarded confident answers. Hallucination is less frequent in recent models, especially when they are trained to express uncertainty, but it has not disappeared, and it is most likely for obscure facts, precise numbers, recent events after the training cutoff, and long outputs. The main defence is grounding: giving the model the relevant sources at query time and asking it to answer from them, with citations a person can check.

Retrieval-augmented generation

Retrieval-augmented generation (RAG) combines a search system with a language model. Before answering, the system retrieves passages relevant to the question from a document collection and inserts them into the prompt. A RAG pipeline has these steps:

1. Ingestion: documents are split into chunks (for example, a few hundred tokens each, with some overlap), and each chunk is indexed.

2. Indexing: chunks are converted to embedding vectors by an embedding model and stored in a vector index, often alongside a keyword index.

3. Retrieval: the query is embedded and the nearest chunks are found (dense retrieval), often combined with keyword search such as BM25 (hybrid retrieval).

4. Reranking: a more accurate cross-encoder model reorders the top candidates.

5. Generation: the best chunks are placed in the prompt with instructions to answer from them and cite sources.

import numpy as np
import re

docs = [
    "Data hall C uses pre-action sprinklers and aspirating smoke detection.",
    "The UPS batteries in room 2 are lithium iron phosphate with off-gas sensors.",
    "Visitors must be escorted at all times and sign the visitor log.",
    "Generator 3 was load tested in March for two hours at 80 percent load.",
    "The cooling units in data hall C were replaced with rear door heat exchangers.",
    "Badge access to data halls requires a fingerprint after 18:00.",
]
query = "what fire protection is in data hall C"


def words(text):
    return re.findall("[a-z0-9]+", text.lower())


vocab = sorted({w for d in docs for w in words(d)})
ix = {w: i for i, w in enumerate(vocab)}
tf = np.zeros((len(docs), len(vocab)))
for r, d in enumerate(docs):
    for w in words(d):
        tf[r, ix[w]] += 1
df = np.count_nonzero(tf, axis=0)
idf = np.log((1 + len(docs)) / (1 + df)) + 1
X = tf * idf
X /= np.linalg.norm(X, axis=1, keepdims=True)

q = np.zeros(len(vocab))
for w in words(query):
    if w in ix:
        q[ix[w]] += idf[ix[w]]
q /= np.linalg.norm(q)
scores = X @ q
for r in np.argsort(scores)[::-1][:3]:
    print(f"{scores[r]:.2f}  {docs[r]}")
NL = chr(10)
top = [docs[r] for r in np.argsort(scores)[::-1][:2]]
prompt = "Answer using only these sources:" + NL + NL.join(f"[{i + 1}] {t}" for i, t in enumerate(top)) + NL + "Question: " + query
print()
print(prompt)

0.45  The cooling units in data hall C were replaced with rear door heat exchangers.
0.39  Data hall C uses pre-action sprinklers and aspirating smoke detection.
0.10  Badge access to data halls requires a fingerprint after 18:00.

Answer using only these sources:
[1] The cooling units in data hall C were replaced with rear door heat exchangers.
[2] Data hall C uses pre-action sprinklers and aspirating smoke detection.
Question: what fire protection is in data hall C

The output shows a typical retrieval failure: the note about cooling units ranks first because it shares many words with the query ("data hall C"), while the note that actually answers the question mentions "sprinklers" and "detection" rather than "fire protection". Dense embeddings, which capture meaning rather than exact words, hybrid search and reranking reduce such errors, and evaluating retrieval separately from generation is essential. If the right passage is not retrieved, the model cannot use it.

RAG is preferred over fine-tuning for adding knowledge because documents can be updated instantly, access controls can be enforced at retrieval time (a user only sees chunks they are permitted to read), and answers can cite sources. Long-context models (with windows of hundreds of thousands to millions of tokens) can sometimes take whole document sets directly, but retrieval remains cheaper and more precise for large collections.

Tool use and agents

Tool use (also called function calling) lets a model call external functions: search engines, calculators, databases, code interpreters, calendars, or any API. The application describes each tool to the model (name, purpose and parameters, usually as a JSON schema). When the model decides a tool would help, it outputs a structured call instead of text; the application runs the tool and returns the result to the model, which continues. Standards such as the Model Context Protocol (MCP), introduced by Anthropic in 2024 and since adopted widely, let tools and data sources be connected to many different AI applications in a uniform way.

import json

inventory = {"H100": 1200, "B200": 640}
power_w = {"H100": 700, "B200": 1000}


def lookup(gpu):
    return {"count": inventory[gpu], "watts": power_w[gpu]}


def calculator(expression):
    allowed = set("0123456789+-*/(). ")
    if not set(expression) <= allowed:
        return "error: invalid characters"
    return round(eval(expression), 3)


tools = {"lookup": lookup, "calculator": calculator}


def fake_model(history):
    last = history[-1]
    if last["role"] == "user":
        return {"tool": "lookup", "args": {"gpu": "B200"}}
    data = json.loads(last["content"])
    if isinstance(data, dict):
        count, watts = data["count"], data["watts"]
        return {"tool": "calculator", "args": {"expression": f"{count} * {watts} / 1000"}}
    return {"answer": f"The B200 fleet draws about {data} kW at full GPU power."}


history = [{"role": "user", "content": "How much power do all our B200 GPUs draw?"}]
for step in range(5):
    action = fake_model(history)
    if "answer" in action:
        print("final:", action["answer"])
        break
    name, args = action["tool"], action["args"]
    result = tools[name](**args)
    print(f"step {step + 1}: {name}({args}) -> {result}")
    history.append({"role": "tool", "content": json.dumps(result)})
print("unsafe input:", calculator("__import__(os).system(rm)"))

step 1: lookup({'gpu': 'B200'}) -> {'count': 640, 'watts': 1000}
step 2: calculator({'expression': '640 * 1000 / 1000'}) -> 640.0
final: The B200 fleet draws about 640.0 kW at full GPU power.
unsafe input: error: invalid characters

An agent is a model running in a loop with tools, deciding on each step what to do next until a goal is reached: researching a question across many web pages, fixing a bug by reading and editing code and running tests, or completing a multi-step business process. Agents can accomplish long tasks but compound errors over many steps, so well-designed agents plan, check their work, ask for confirmation before consequential actions, and operate with limited permissions.

Security risks

Connecting models to data and tools creates new risks:

• Prompt injection: instructions hidden in content the model reads (a web page, email or document) can hijack its behaviour, for example telling it to send private data to an attacker. Because models do not reliably distinguish data from instructions, defences combine limited permissions, separation of untrusted content, human confirmation of sensitive actions, output filtering and monitoring.
• Data leakage: models may reveal confidential information present in their context, or retrieval may surface documents a user should not see unless access control is enforced.
• Unsafe actions: tools that send messages, spend money or modify systems need strict authorization, as in the calculator example, where untrusted input must never reach a general code evaluator.
• Insecure output handling: model output inserted into web pages, SQL queries or shell commands must be treated as untrusted input.

The OWASP Top 10 for LLM Applications is a widely used checklist of these risks.

Evaluation and cost

LLM applications need systematic evaluation: test sets of realistic inputs with expected outputs or grading rubrics, automated scoring (including using a model as a judge, checked against human ratings), and monitoring in production. Cost is driven by tokens: API prices are quoted per million input and output tokens, and output tokens cost more. Long prompts, retrieved documents and reasoning tokens all add cost and latency, so caching, smaller models for simple steps, and routing between models are common optimizations.

Regulation of generative AI

Canada does not yet have a comprehensive AI law. The proposed Artificial Intelligence and Data Act died when Parliament was prorogued in January 2025. Since then the federal government, through its first Minister of Artificial Intelligence and Digital Innovation, has pursued a national AI strategy, completed a public consultation on AI transparency and safety, and introduced bills on privacy and consumer data and on online harms, including deepfakes, while saying that rules for AI deployment are still being considered. A voluntary code of conduct for generative AI developers has been in place since 2023. Existing laws on privacy (PIPEDA and provincial laws such as Quebec's Law 25, which requires notice of automated decisions), human rights, consumer protection and copyright already apply to AI systems.

Organizations deploying LLMs internationally must also consider the European Union's AI Act, whose obligations for general-purpose AI models began applying in August 2025. Good practice, whatever the law, includes documenting systems, testing for bias and errors, disclosing AI use to users, and keeping humans accountable for consequential decisions.

Lesson 8 explains where the cost and speed of generation come from, and Lesson 10 looks more closely at the embeddings and vector search behind retrieval.