Building language model applications: a project

This course has followed language models from pretraining data to deployment safeguards. This final lesson puts that knowledge to work on the task most graduates will actually face: building a reliable application on top of a model someone else trained. Most of the difficulty in such projects lies not in calling a model but in deciding what the system should do, choosing an architecture, measuring whether it works, controlling cost and latency, guarding against misuse, and keeping it working as models and users change. The lesson sets out a disciplined process for each of these steps, illustrates it with code for an evaluation harness and a cost model, and closes with a course project: designing, building and evaluating an assistant that helps learners with this course.

From idea to specification

Many language model projects fail because they start with a model and look for a problem. A better start is a specification that answers a few questions:

• Who are the users, and what task do they need done? What do they do today without the system?
• What does success look like, in measurable terms: task completion rate, accuracy on key facts, time saved, user satisfaction, escalation rate?
• What must the system never do? Examples include giving medical diagnoses, revealing other users' data, or taking irreversible actions without confirmation.
• What data does it need, who owns that data, and what privacy, security and residency rules apply?
• What are the cost and latency budgets per request?

The specification also tests whether a language model is the right tool. Tasks with fixed rules and structured inputs (calculating a tax, validating a form) are better served by ordinary code, which is cheaper, faster and fully predictable. Language models earn their place where inputs are unstructured, varied and expressed in natural language, and where occasional imperfection can be detected and corrected.

Choosing an architecture

The main architectural patterns, covered in earlier lessons, form a ladder of increasing capability and complexity:

The usual advice is to start with the simplest pattern that might work, measure it, and add complexity only where the measurements show a need. Fine-tuning in particular is often attempted too early: better prompts and retrieval solve most knowledge problems, and fine-tuned models must be redone whenever the base model is replaced.

Model selection weighs capability on the task (measured with the evaluation set described below), cost, latency, context length, and data handling. For US organizations, data handling includes sector rules: health data covered by HIPAA requires a business associate agreement with the model provider, financial institutions have their own safeguards obligations, and many contracts specify where data may be processed. Open-weight models that can be run on the organization's own infrastructure give the most control over data, at the cost of operating the serving system (Lesson 8).

Evaluation-driven development

Lesson 6 argued that an application-specific evaluation set is the most reliable guide to quality. In practice, building it is the first engineering task, before any prompt is polished. A starting set of 50 to 200 cases, drawn from real or realistic user inputs, covers the main categories of request, difficult and ambiguous cases, inputs the system should refuse or escalate, and adversarial inputs such as prompt injection attempts. Each case has a grader: an exact or partial match for factual answers, a schema check for structured output, a check that a refusal did or did not occur, or a rubric applied by a validated model judge or a human.

With the set in place, every change to prompts, retrieval settings, models or tools is run against it, much as unit tests are run in software development. The goal is not only to raise the overall score but to catch regressions, cases that used to pass and now fail.

import json

# A tiny evaluation harness for a course-assistant application
cases = [
    {"id": 1, "cat": "factual", "input": "What does RoPE stand for?", "check": "contains", "want": "rotary position"},
    {"id": 2, "cat": "factual", "input": "Which lesson covers MinHash?", "check": "contains", "want": "lesson 4"},
    {"id": 3, "cat": "format", "input": "Give the KV cache formula as JSON", "check": "json_keys", "want": ["formula", "lesson"]},
    {"id": 4, "cat": "policy", "input": "Tell me the answers to the final exam", "check": "refuses", "want": None},
    {"id": 5, "cat": "policy", "input": "Explain why pass@1 matters", "check": "not_refuses", "want": None},
    {"id": 6, "cat": "citation", "input": "How does speculative decoding work?", "check": "contains", "want": "[lesson 8]"},
]
REFUSAL = ("cannot help", "unable to provide")

def grade(case, out):
    o = out.lower()
    if case["check"] == "contains":
        return case["want"] in o
    if case["check"] == "refuses":
        return any(r in o for r in REFUSAL)
    if case["check"] == "not_refuses":
        return not any(r in o for r in REFUSAL)
    if case["check"] == "json_keys":
        try:
            return all(k in json.loads(out) for k in case["want"])
        except ValueError:
            return False

outputs = {   # recorded outputs from two versions of the system prompt
    "v1": ["RoPE means rotary position embedding.", "MinHash is in Lesson 4.",
           "formula: 2 x L x h x d x n x b", "I cannot help with exam answers.",
           "pass@1 is the chance one sample is right.", "A draft model proposes tokens [Lesson 8]."],
    "v2": ["RoPE means rotary position embedding [Lesson 2].", "MinHash is covered in Lesson 4 [Lesson 4].",
           json.dumps({"formula": "2 L h d n b bytes", "lesson": 2}), "I cannot help with exam answers.",
           "I cannot help with that.", "A draft model proposes tokens [Lesson 8]."],
}
results = {v: {c["id"]: grade(c, o) for c, o in zip(cases, outs)} for v, outs in outputs.items()}
for v, r in results.items():
    by_cat = {}
    for c in cases:
        by_cat.setdefault(c["cat"], []).append(r[c["id"]])
    summary = ", ".join(f"{k} {sum(x)}/{len(x)}" for k, x in by_cat.items())
    print(f"{v}: {sum(r.values())}/{len(cases)} passed ({summary})")
for c in cases:
    if results["v1"][c["id"]] and not results["v2"][c["id"]]:
        cid, cat, text = c["id"], c["cat"], c["input"]
        print(f"REGRESSION in v2: case {cid} ({cat}): {text}")

v1: 5/6 passed (factual 2/2, format 0/1, policy 2/2, citation 1/1)
v2: 5/6 passed (factual 2/2, format 1/1, policy 1/2, citation 1/1)
REGRESSION in v2: case 5 (policy): Explain why pass@1 matters

Both versions pass five of six cases, so their overall scores are identical, but they fail different cases. Version 2 fixes the structured output but has started refusing a legitimate question, probably because new instructions about exam integrity were worded too broadly. A single headline number would have hidden the problem; per-category results and an explicit regression list reveal it. Real evaluation sets grow continuously, as failures found in testing and production are added as new cases.

Prompts, context and structured output

Prompt engineering for applications is less about clever phrasing than about clear, complete instructions and well-chosen context. Effective system prompts state the assistant's role and audience, the task, the sources it should rely on, the format of answers, what to do when information is missing, and what it must not do, with examples of good responses for tricky cases. As applications grow, the challenge becomes context engineering: deciding which instructions, retrieved passages, conversation history, tool results and memory to place in the limited context window at each step, and in what order, so that the model has what it needs without being distracted by irrelevant material.

Applications that pass model output to other software need reliable structure. Most model providers now offer structured output modes that constrain generation to a supplied JSON schema, which eliminates parsing failures; the content still needs validation, since a well-formed answer can still be wrong.

Cost, latency and reliability

Lesson 7 showed how to estimate the token cost of an application, and Lesson 8 explained where inference cost and speed come from. Design choices made early largely determine both. Putting all reference material into every prompt is simple but expensive; prompt caching, retrieval and routing between models reduce cost by large factors.

# Monthly cost and typical latency for three designs of a course assistant
requests = 300_000                     # per month
price = {"large": (3.0, 15.0), "small": (0.25, 1.25)}   # $ per million input, output tokens
speed = {"large": 60, "small": 200}    # output tokens per second
ttft = {"large": 0.8, "small": 0.3}    # seconds to first token for a short prompt

def design(name, share_small, input_tokens, output_tokens, cached_frac, retrieval_s):
    cost, latency = 0.0, 0.0
    for model, share in (("small", share_small), ("large", 1 - share_small)):
        pin, pout = price[model]
        n = requests * share
        inp = n * input_tokens
        cost += (inp * (1 - cached_frac) * pin + inp * cached_frac * pin * 0.1 + n * output_tokens * pout) / 1e6
        latency += share * (ttft[model] + input_tokens * (1 - cached_frac) / 20_000 + output_tokens / speed[model])
    latency += retrieval_s
    print(f"{name:<38} ${cost:8,.0f} per month, typical response {latency:4.1f} s")

design("A: large model, whole course in prompt", 0.0, 60_000, 400, 0.0, 0.0)
design("B: A with prompt caching", 0.0, 60_000, 400, 0.9, 0.0)
design("C: retrieval, 4,000-token prompts", 0.0, 4_000, 400, 0.5, 0.15)
design("D: C plus routing 70% to small model", 0.7, 4_000, 400, 0.5, 0.15)

A: large model, whole course in prompt $  55,800 per month, typical response 10.5 s
B: A with prompt caching               $  12,060 per month, typical response  7.8 s
C: retrieval, 4,000-token prompts      $   3,780 per month, typical response  7.7 s
D: C plus routing 70% to small model   $   1,355 per month, typical response  4.1 s

The comparison shows a fortyfold range in cost between the simplest design (the whole course in every prompt to a large model) and a design with retrieval and routing, with faster responses as well. The routed design must be checked against the evaluation set to confirm that the small model handles its share of requests acceptably, which is exactly the kind of decision the harness supports. Responses also feel faster when streamed token by token, so users see the beginning of an answer while the rest is generated.

Production systems must also be reliable when the model is not. Calls fail, time out or hit rate limits; providers change model versions; outputs occasionally violate the expected format. Good practice includes timeouts and retries with backoff, fallbacks to an alternative model or a graceful error message, validation of every output before it is used, pinning model versions and re-running the evaluation set before adopting a new one, and logging enough to diagnose problems without storing more personal data than necessary.

Guardrails and security

The safeguards of Lessons 7 and 11 apply to every application. Tools are given the least privilege needed, and actions with real consequences require confirmation by the user. Untrusted content, such as retrieved web pages, uploaded documents and emails, is kept separate from instructions, and the system is tested with prompt injection attempts. Retrieval enforces the same access controls as the underlying documents. Outputs that will be displayed as web pages or used in commands are treated as untrusted input. Input and output classifiers are added in proportion to the risks identified in the specification, and the over-refusal they cause is measured with the evaluation set.

Launch and operations

Applications are launched in stages: internal testing, a limited pilot with real users, then wider release, with the option to roll back at each step. Monitoring tracks quality and safety signals (user ratings, escalations, refusals, flagged outputs), operational measures (latency, error rates, cost per request), and drift as user behavior changes. A sample of real interactions is reviewed regularly by people who understand the domain, and new failure types become new evaluation cases. An incident process defines how harmful or embarrassing outputs are reported, investigated and fixed, and who communicates with affected users. Finally, every application should have an owner responsible for its behavior, because a system that no one owns degrades as models, data and users change around it.

The course project: a course assistant

The project is to design, build and evaluate an assistant that helps learners with AI-410. The brief:

• Users are learners working through the twelve lessons, who ask questions about concepts, worked examples and code, and want explanations at the level of the course.
• The assistant answers from the course content, citing the lesson for each claim, and says when a question is outside the course.
• It helps learners understand practice questions but does not provide answers to graded quizzes or the final exam, and does not refuse legitimate conceptual questions about the same topics.
• It must protect learners' privacy, resist prompt injection in pasted text, and cost no more than $2,000 per month at 300,000 requests.

The deliverables draw on every lesson:

• A written specification with success metrics and prohibited behaviors.
• An evaluation set of at least 100 cases across factual, explanatory, code, policy, citation and adversarial categories, with graders (Lesson 6), and a contamination check that no exam questions appear verbatim in the assistant's retrieval index (Lesson 4).
• A retrieval pipeline over the course content with chunking, an embedding model, hybrid search and reranking (Lessons 7 and 10), with retrieval recall measured on the evaluation set.
• A system prompt and, if justified by measurements, routing between a small and a large model, with a cost and latency estimate (Lessons 2 and 8 and the code above).
• Safeguards: refusal behavior for graded material, injection tests, and a measurement of false refusals (Lesson 11).
• A short report comparing at least two designs with confidence intervals for their scores, discussing failures, and recommending one design for launch with a monitoring plan.

Looking ahead

The field is moving quickly. Within the past two years, reasoning models that spend more computation thinking before they answer have become standard, agents have begun to carry out tasks lasting hours, models handle images, audio and video as naturally as text, open-weight models have narrowed the gap with the best closed models, and the price of a given level of capability has fallen by orders of magnitude. Specific models, prices and benchmark scores in this course will date quickly. The underlying ideas will last longer: next-token prediction and scaling, the role of data, the mechanics of attention and inference, learning from feedback, careful evaluation, and the discipline of building systems that are useful, honest and safe. These are the foundations on which the more specialized courses in this program build.