Piloting and scaling AI initiatives

Noor had seen many AI pilots succeed and then go nowhere. "They call it pilot purgatory," she told Andre. "The pilot works, everyone says it's promising, and a year later it's still running at the same four sites because nobody decided what would happen next." She wanted the no-show pilot planned so that, at the end of twelve weeks, there would be no doubt about what to do.

This lesson covers how to design a pilot that answers the right questions, how to turn its results into a decision to scale, pause or stop, and what changes when an AI solution moves from a few sites to the whole organization. It connects the exam guide's skills on prioritization (scale, pause or terminate), leading indicators and transitions.

What a pilot is for

A pilot is a limited, real-world trial of a solution, designed to answer specific questions before a larger commitment. It is different from a proof of concept, which tests whether something can work at all, usually on historical data or in a lab. The no-show model had already passed a proof of concept in the autumn assessment, when it reached a recall of 0.62 on a test set. The pilot asks the questions a test set cannot answer:

• Do clients actually respond to the extra reminders, and do enough freed slots get refilled?
• Do reception staff follow up on the flags consistently, and does the workload fit their day?
• Does the model perform as well on new bookings as it did on old ones, at every clinic?
• Do any unexpected problems appear, such as complaints, confusion or unfair effects?
• What does it really cost to run, including staff time?

A pilot that is not designed to answer questions like these is really a soft launch, and soft launches are how pilot purgatory begins.

Designing the pilot

Including a rural clinic was deliberate. The autumn assessment had found lower recall at two rural clinics. A pilot made up only of large city clinics might have looked better and told the board less.

Decision gates

A decision gate is a planned point where evidence is compared against criteria agreed in advance, and a decision is made: proceed, adjust, pause or stop. Gates are what turn a pilot into a decision. Andre built the gate criteria from the business case in lesson 4, where the break-even recovery rate translated into about 680 recovered appointments a year, or about 0.5 fewer no-shows per 100 appointments.

The middle column matters as much as the outer ones. Most pilots land somewhere in between: promising but not conclusive. Deciding in advance what "adjust and extend" means, such as four more weeks with an earlier daily list for reception, stops a middling result from drifting into an open-ended trial. Each extension should have its own end date and its own gate.

Applying a gate often produces a split decision like this one. That is a strength, not a weakness: it lets the organization capture value where the evidence is strong while fixing problems where it is not.

Leading indicators during the pilot

Gates come at the end, but leading indicators, introduced in lesson 5, are watched every week. They let the team fix problems while the pilot is still running. If adoption at one clinic is low in week two, the cause might be a missing step in the morning routine, not a weakness in the model. Noor set up a one-page weekly summary showing adoption, scores delivered on time, model performance and complaints for each clinic, and she and Andre spent fifteen minutes on it every Monday.

From pilot to scale

Scaling is more than switching the solution on everywhere. What works at four sites with close attention from the project team must now work at eleven, run by people who were not involved in the pilot. Several things change:

The model operations row describes what practitioners call MLOps: the processes for deploying, monitoring, retraining and versioning models reliably. A strategist does not need to run MLOps, but needs to budget for it and to make sure someone owns it. A model that is not retrained will slowly drift as client behaviour changes, and its value will quietly fade.

Rolling out in waves

Rather than switching all clinics on at once, Andre planned to roll out in waves. The first wave would add the three remaining city clinics in May 2027; the second, after the rural fix had been checked, would add the remaining rural clinics. Waves limit the damage if something goes wrong, let support capacity grow gradually, and give each wave's lessons to the next. They also keep a natural comparison group for a few more weeks, which strengthens the measurement of effect.

Transitions and business continuity

Scaling an AI solution changes how work is done, so it is also a transition. The considerations from lesson 3 apply again: business continuity, cost implications, data readiness and performance impact. For continuity, each clinic keeps the standard single reminder running automatically, so if the scoring pipeline fails, no client misses their normal reminder; only the extra reminder is lost. For cost, the jump from four clinics to eleven roughly triples reception follow-up time, which must fit into staffing. For data, newer clinics with shorter history need checks before they join. For performance, the daily list must still arrive by 7:30 a.m. when it covers three times as many bookings.

When to stop

Stopping a pilot is a legitimate outcome, and a well-designed gate makes it easier. If the effect is below break-even, the business case does not hold, and continuing means spending money to lose money. If a stop condition is triggered, such as evidence that flags were being used to delay bookings, the pilot should stop at once while the cause is fixed, regardless of financial results.

A stopped pilot still produces value if its lessons are recorded: what was tried, what happened, why it was stopped, and what would need to be true for the idea to be tried again. Noor kept a short file of such records from previous jobs. Several of the ideas in it had succeeded years later, when data or technology had caught up.

Common reasons pilots fail to scale

Noor kept a list of the reasons she had seen AI pilots stall, drawn from earlier jobs and from conversations with peers. Few of them were about the model:

• No owner for the scaled solution. The project team moved on and nobody in the business took responsibility.
• Workflow not redesigned. The AI output arrived, but nobody had changed the process to act on it, so it was ignored.
• Success criteria never agreed. Everyone had a different idea of what good looked like, so no decision could be made.
• Costs at scale not planned. The pilot ran on a project budget; the operating budget had no line for it.
• Trust lost early. A few visible mistakes in the first weeks, without explanation or fixes, made users stop relying on the tool.
• Data that worked for the pilot sites did not exist elsewhere. The model could not be extended without new data work.

Each item has a matching remedy in this lesson: named owners, process changes designed alongside the model, gate criteria agreed in advance, a scaled cost plan, quick responses to early problems, and data checks for each new site. Andre added the list to the pilot plan as a set of risks to watch, with a name against each.

Andre also drafted a one-page pilot charter for every pilot clinic manager to sign. It stated the pilot's purpose, what would change for staff and clients, what would not change, who to contact with problems, the dates of the weekly check-ins and the end-of-pilot gate, and the conditions under which the pilot would stop early. Signing it is not a formality. It makes the clinic managers partners in the pilot rather than hosts of someone else's experiment, and when Andre walked two of them through the draft, both suggested improvements to the daily list.

Noor added one more practice from her previous work: a short written review at the end of every pilot, whatever its outcome. The review recorded the gate results, what had surprised the team, what they would do differently next time, and which assumptions in the business case had held. Reviews like these become an organization's memory for AI, so that the next pilot, for lab-report extraction, can start from what the first one learned rather than from scratch.