Building and evaluating systems with large language models

Many capstone projects now build on large language models (LLMs) rather than training models from scratch: assistants that answer questions from a document collection, tools that extract structured data from text, agents that complete multi-step tasks, or classifiers built from prompts. These systems can reach useful performance quickly, but they bring their own engineering and evaluation challenges. Outputs are open-ended and hard to score automatically; behaviour can change with small prompt edits or model updates; models can state falsehoods confidently; and costs and latency depend on token counts. This lesson covers choosing between prompting, retrieval and fine-tuning, building retrieval-augmented generation (RAG) pipelines, evaluating each component and the whole system, building evaluation sets, using models as judges carefully, structured outputs and tool use, and managing cost, latency and change.

Prompting, retrieval or fine-tuning?

Three broad ways to adapt an LLM to a task are commonly combined:

• Prompting: instructions, examples (few-shot prompting) and formatting constraints in the input. Fast to iterate and requires no training, but limited by the model's existing knowledge and the context window.
• Retrieval-augmented generation: retrieving relevant passages from a document collection and including them in the prompt, so that answers are grounded in specific, current sources and can cite them. Suited to question answering over organizational documents.
• Fine-tuning: further training of a model on task examples, which can improve consistency, style and performance on specialized formats, and allow a smaller, cheaper model to match a larger one on a narrow task. It needs labelled data, compute and careful evaluation.

A sensible capstone path is to start with prompting a capable model to establish a strong baseline, add retrieval if the task needs specific knowledge, and consider fine-tuning only when evaluation shows a persistent gap that examples could close, or when cost and latency require a smaller model.

The RAG pipeline

A typical RAG system has several stages, each of which can fail:

• Ingestion: documents are collected, cleaned, and split into chunks of a few hundred tokens, with metadata such as source, date and section.
• Indexing: chunks are represented for search, using sparse methods such as TF-IDF or BM25, which match words, dense embeddings from neural models, which match meaning, or both (hybrid search).
• Retrieval: for each query, the top \(k\) chunks are retrieved, sometimes re-ranked by a more accurate model.
• Generation: the LLM receives the question and retrieved chunks with instructions to answer only from the sources, cite them, and say when the answer is not found.
• Post-processing: checking citations, formatting and applying safety filters.

Evaluating retrieval

Because the generator can only use what is retrieved, retrieval quality sets an upper bound on answer quality. Evaluate it separately with a set of questions labelled with the passages that contain their answers, using recall at \(k\) (the share of questions whose relevant passage appears in the top \(k\)) and mean reciprocal rank. Low retrieval recall points to problems in chunking, indexing, query formulation or vocabulary mismatch, where users phrase questions differently from the documents.

import numpy as np
# A tiny retrieval system for RAG: TF-IDF vectors and recall at k
docs = ["refund requests must be filed within 30 days of purchase",
        "shipping to remote regions takes up to ten business days",
        "warranty covers manufacturing defects for one year",
        "to reset your password use the account settings page",
        "refunds are paid to the original payment method",
        "international shipping requires a customs declaration"]
queries = [("how long do I have to ask for a refund", 0), ("how is my refund paid", 4),
           ("I forgot my password", 3), ("does the warranty cover defects", 2), ("customs for shipping abroad", 5), ("can I get my money back", 0)]
def tokens(t):
    return [w for w in t.lower().split() if len(w) > 2]
vocab = sorted({w for d in docs for w in tokens(d)})
index = {w: i for i, w in enumerate(vocab)}
tf = np.zeros((len(docs), len(vocab)))
for i, d in enumerate(docs):
    for w in tokens(d): tf[i, index[w]] += 1
idf = np.log(len(docs) / (1 + (tf > 0).sum(0))) + 1
D = tf * idf; D /= np.linalg.norm(D, axis=1, keepdims=True)
hits = {1: 0, 2: 0}
for q, gold in queries:
    v = np.zeros(len(vocab))
    for w in tokens(q):
        if w in index: v[index[w]] += idf[index[w]]
    if np.linalg.norm(v) == 0:
        print(f"{q:40s} -> no shared words, nothing retrieved (gold {gold})")
        continue
    v /= np.linalg.norm(v)
    ranking = np.argsort(-D @ v)
    for k in hits: hits[k] += int(gold in ranking[:k])
    print(f"{q:40s} -> top doc {ranking[0]} (gold {gold})")
for k, h in hits.items():
    print(f"recall@{k}: {h / len(queries):.2f}")

how long do I have to ask for a refund   -> top doc 0 (gold 0)
how is my refund paid                    -> top doc 4 (gold 4)
I forgot my password                     -> top doc 3 (gold 3)
does the warranty cover defects          -> top doc 2 (gold 2)
customs for shipping abroad              -> top doc 5 (gold 5)
can I get my money back                  -> no shared words, nothing retrieved (gold 0)
recall@1: 0.83
recall@2: 0.83

The word-matching retriever finds the right document for five of six questions, but fails on "can I get my money back", which shares no words with the refund policy. This vocabulary mismatch is the classic weakness of sparse retrieval; dense embeddings, query rewriting by the LLM, or hybrid search address it. Evaluating retrieval separately reveals such failures before they appear as wrong or unsupported answers.

Evaluating generated answers

Generated answers are judged on several dimensions: correctness (is the answer right?), completeness (does it cover what was asked?), faithfulness or groundedness (is every claim supported by the retrieved sources, or did the model add unsupported content?), citation accuracy, appropriate refusal (does it decline when the answer is not in the sources?), tone and format, and safety. Methods include:

• Reference-based checks: comparing answers with gold answers by exact match for short factual answers, or by checking that key facts are present.
• Human evaluation: domain experts or trained raters scoring answers with a rubric, the most reliable method but slow and expensive.
• Model-based evaluation ("LLM as judge"): a capable model scores answers against a rubric. It scales well but can be biased, for example favouring longer answers, answers in its own style or the first option shown, so it must be validated against human ratings on a sample before being trusted.
• Programmatic checks: valid JSON or schema compliance, presence of citations, length limits and banned content.

Building an evaluation set

An LLM system needs an evaluation set as much as a classifier needs a test set. A good set contains realistic questions or inputs, ideally collected from real users or the partner rather than invented by the team; reference answers or relevant passages; coverage of important categories, including rare but critical ones; unanswerable and adversarial inputs, such as questions outside the documents, ambiguous requests and attempts to make the system ignore its instructions; and enough items, often 100 to 500, to detect meaningful differences. Keep part of it as a held-out test set, used only for final evaluation, so that prompt tuning does not overfit the whole set.

Structured outputs, tools and agents

Many applications need outputs in a structured form, such as JSON matching a schema for extracted fields, or function calls that invoke tools like search, calculators or databases. Modern LLM APIs support constrained or schema-guided output and tool use, which greatly improves reliability. Systems that let a model plan and call tools over several steps, often called agents (AI-470), can complete complex tasks but multiply failure points: each step can go wrong, and errors compound. Evaluate agents on complete tasks, measuring success rate, the number of steps and tool calls, cost and the frequency of unsafe or irreversible actions, and keep humans in the loop for consequential actions.

Cost, latency and change

LLM systems are priced and timed by tokens. Costs depend on the number of input tokens, including retrieved context and instructions, and output tokens; latency depends on model size, output length and provider load. Techniques to manage both include choosing the smallest model that meets the quality target, shortening prompts, retrieving fewer but better chunks, caching responses to repeated questions, and batching offline tasks (Lesson 11). Models accessed through APIs may be updated or retired by the provider, changing behaviour; record model versions, pin them where possible, and rerun the evaluation set whenever the model, prompt or retrieval changes. A version-controlled prompt with an automated evaluation run is the LLM equivalent of a tested code change.

Risks specific to LLM systems

LLM systems carry distinctive risks that the safety review (Lesson 9) must address: hallucination, stating plausible but false information; prompt injection, where text in a document or user input instructs the model to ignore its rules or reveal data; leakage of confidential data through prompts sent to external providers or through outputs; harmful or biased content; over-reliance by users who trust fluent answers; and copyright concerns in generated content. Grounding answers in sources with citations, clear user interface messages about limitations, input and output filtering, access controls, and human review for important uses all reduce these risks.

Prompt design practices

Prompts are a form of code and benefit from the same discipline. Good practices include stating the task, the audience and the required format explicitly; giving the model the information it needs rather than relying on its general knowledge; including a few representative examples for complex formats; telling the model what to do when information is missing, such as answering "not found in the documents"; separating instructions from untrusted input text with clear delimiters; and keeping prompts under version control with a changelog. Small, controlled changes evaluated on the development set are more reliable than large rewrites.

Evaluating extraction and classification with LLMs

When an LLM performs a structured task, such as extracting fields or assigning categories, evaluation can be as rigorous as for a conventional model: compare outputs with gold labels using exact match, precision and recall per field or class, confusion matrices and slice analysis. Check schema validity separately, and measure consistency by running the same input several times, since sampling can produce different outputs. Low-temperature settings and constrained output formats improve consistency.

As with any component, keep an evaluation dashboard for the LLM pipeline that shows retrieval recall, answer quality, faithfulness, refusal accuracy, cost and latency for each configuration, so that the team can see at a glance whether a change helped and what it cost.