Scaling from pilot to production

The product description pilot ended its eight weeks on November 6, 2026 with results that met the decision rule from lesson 2: median editing time of 17 minutes, editor quality ratings level with the baseline, factual errors down to about one in every 100 drafts after the data fixes, and 41 of the 50 most recent descriptions started from an AI draft. On November 10 the steering committee voted to expand it to every new product from January 2027. Jules is delighted, and Farah is cautious in the way experienced architects are. "Pilots succeed because a small, motivated team watches everything," she says. "Production has to succeed when nobody is watching closely, at ten times the volume, on a model that the provider will upgrade at some point next year." This lesson covers the gap between a successful pilot and a reliable production system, and the plan Birchline writes to cross it.

Why pilots stall

Many organizations have far more generative AI pilots than production systems. The reasons are familiar from other technology but sharper with AI.

A staged rollout

Birchline does not switch from eight copywriters to every product overnight. The production plan expands in stages, each with a short check before the next.

• Stage 1, January 2027: all new products in two categories (footwear and tents), the most complex specifications, with the full review screen and daily quality sampling.
• Stage 2, February: all new products, with sampling reduced to a weekly audit if quality holds.
• Stage 3, spring: refreshing old descriptions, about 9,000 products, run in batches with stricter checks because old spec sheets are less reliable.

Each stage has a rollback plan: if quality drops below the agreed level, the system can be switched off for a category in minutes and copywriters return to writing from scratch, which they can still do. Keeping the old path available until the new one is proven is one of the simplest risk controls in any rollout.

Evaluation at production scale

In the pilot, editors effectively evaluated every draft. In production, Birchline needs an evaluation set: a fixed collection of test cases with known good answers or criteria, run whenever anything changes. The description team builds a set of 300 products, chosen to include every category, both languages, unusual sizing systems, missing fields and spec sheets known to contain errors. Each case has the expected facts and a rubric score from an editor.

# Compare two prompt versions on the same evaluation set (illustrative results).
eval_results = {
    "prompt v3": {"facts_correct": 291, "missing_flagged": 18, "missing_total": 20, "avg_rubric": 4.3},
    "prompt v4": {"facts_correct": 294, "missing_flagged": 15, "missing_total": 20, "avg_rubric": 4.5},
}
cases = 300
for name, r in eval_results.items():
    fact_rate = r["facts_correct"] / cases
    miss_recall = r["missing_flagged"] / r["missing_total"]
    print(f"{name}: facts {fact_rate:.3f}, missing-field recall {miss_recall:.2f}, rubric {r['avg_rubric']}")

prompt v3: facts 0.970, missing-field recall 0.90, rubric 4.3
prompt v4: facts 0.980, missing-field recall 0.75, rubric 4.5

The new prompt version scores better on facts and rubric but flags fewer missing fields, which is the behaviour that protects against invented specifications. Farah's rule is that no change ships if it makes a safety-critical measure worse, even if the average improves, so prompt v4 goes back for another revision. Without the evaluation set, the team would have seen only the higher rubric score and shipped it.

Monitoring in production

Once in production, a generative AI system needs monitoring like any service, plus some AI-specific signals.

Drift deserves explanation. A system can degrade without any change to its code. The inputs may change: a new supplier sends spec sheets in a different format, or a new product category arrives. The world may change: a new regulation affects how products may be described. Or the model may change: providers improve models and eventually retire older versions. Monitoring compares current inputs and outputs with what was tested, so the team notices drift before customers do.

Managing model and prompt versions

Production systems should know exactly which model version, prompt version and settings produced each output. Birchline keeps prompts in version control alongside the application code, records the model version in each output's log entry, and pins a specific model version rather than always using "the latest". When the provider announces a new version, the team runs the evaluation set on it, compares the results, and switches only when the new version is at least as good on every critical measure. Google Cloud, like other providers, publishes model version and retirement information, and Birchline's AI inventory from lesson 9 shows which systems use which model, so nothing is missed.

This discipline has a name in industry, often MLOps or, for large language models, LLMOps: the practices of deploying, monitoring, versioning and improving machine learning systems reliably. Leaders do not need the tools, but they should expect the practices.

Costs at scale

Usage costs that were trivial in a pilot can grow quickly in production, particularly for customer-facing systems where volume depends on customers. Jules's cost forecast for the shopping assistant, planned for spring 2027, models three volumes, because nobody knows how many shoppers will use it.

# Illustrative monthly usage forecast for the shopping assistant. Not real prices.
price_per_million_tokens = 2.50
tokens_per_conversation = 9000
for conversations in (20_000, 60_000, 150_000):
    usage = conversations * tokens_per_conversation / 1_000_000 * price_per_million_tokens
    print(f"{conversations:>7,} conversations a month -> about {usage:>6,.0f} dollars of model usage")

 20,000 conversations a month -> about    450 dollars of model usage
 60,000 conversations a month -> about  1,350 dollars of model usage
150,000 conversations a month -> about  3,375 dollars of model usage

Cost controls at scale include setting budgets with alerts on the project, limiting the length of conversations and responses, caching answers to common questions, routing simple requests to a smaller model, and keeping grounding passages short and relevant. Each lever affects quality too, so changes go through the evaluation set like any other change.

Operating model: who runs it

A production system needs people. Birchline's operating model for each production AI system names the business owner, who decides what the system should do and answers for its outcomes; the technical owner, who keeps it running and manages versions; the reviewers, whose time is budgeted; and the support path for users. The AI review group receives a monthly summary of quality, incidents and cost for every high and medium-risk system.

As more systems reach production, Birchline sets up a small AI centre of excellence, building on the cloud centre of excellence from GCP-102: Farah, Jules, an analyst, and part-time representatives from security and data. It maintains shared assets: the prompt library, evaluation sets, the review screen, the monitoring dashboards and the procurement questions. Shared assets are how a second and third production system get built in half the time of the first.

Readiness checklist before each stage

Before each rollout stage, the technical owner completes a short readiness checklist, and the business owner signs it. The checklist makes the gates in the figure concrete.

The checklist is short on purpose. Each line has caught a real gap at least once in Birchline's experience or in Farah's previous work. The rollback line, for example, exists because at her previous employer a tool could only be switched off by one engineer, who was on holiday the week it needed switching off.

Learning from production

Production also produces knowledge that improves the next system. Birchline keeps a short running log for each production system with notable failures, user feedback themes, cost surprises and the fixes made. Once a quarter, the centre of excellence reads across the logs for patterns. In the description tool's first planned quarter, Farah expects patterns around older spec sheets and new categories, and the log is designed to capture them so that the policy assistant and the shopping assistant can avoid repeating them.

A final production habit is to retire systems deliberately. If a use case stops delivering value, or a better option appears, the owner proposes retirement to the review group, users are told in advance, data and logs are kept or deleted according to policy, and the AI inventory is updated. Systems that are never retired accumulate cost and risk long after anyone remembers why they exist.