MSBA-303 · Lesson 12 of 12 · 20 min

Case study: rolling out Copilot to a team

▶ Video coming soon A video version of this lesson is on the way.

The last session, on July 21, 2027, brought the series together. The four-seat pilot had run for nine weeks, since May 17, and the owners wanted a recommendation: expand Copilot to eight people in September, keep it at four, or stop. Rosa, Priya and Dmitri prepared the recommendation in the Portland boardroom as a working session, using everything from the course. This lesson follows that case study step by step, from goals and pilot results to plan, policies, budget, training and review, and ends with the rollout plan the owners approved. Dates from August onward are plans, not results.

Step 1: goals and success measures

A rollout without goals cannot be judged. Back in May, before the pilot, Priya and Rosa had written three goals and how each would be measured. Writing them first stopped the team from choosing flattering numbers afterwards.

Pilot goals and measures, written before the pilot started.
GoalMeasurePilot target
Developers spend less time on routine codeSelf-reported weekly estimate, plus pull request time to mergeA noticeable fall without more defects
Code quality does not dropBugs found after release; review comments per pull requestNo increase
Data stays protectedNo secrets or personal data in prompts; exclusions workingZero incidents
Cost stays predictableMonthly bill against budgetExtra usage under 20 US dollars a month

Step 2: what the pilot showed

Dmitri assembled the results from the usage metrics, the billing pages, the team's review-comment tracking from lesson 6 and a short survey.

Pilot results.
MeasureBefore pilotPilot (9 weeks)
Average daily active usersNot applicable3.2 of 4 seats
Acceptance rate of suggestionsNot applicable30 in 100
Median pull request time to merge26 hours19 hours
Bugs found after release, per month33
Copilot review comments appliedNot applicable45 in 100
Data incidents00
Extra usage per monthNot applicable9 to 13 US dollars
Self-reported time saved per developer per weekNot applicableAbout 2 hours

The team was careful about what the results proved. Faster merges came alongside Copilot code review catching simple problems, but the pilot months were also quieter than spring. Bugs did not rise, which met the quality goal. The time saved was self-reported, which tends to overstate. Priya summarized it honestly: "Useful, safe and affordable at our size, with some of the benefit uncertain."

Step 3: the case for expansion

Four more people had asked for seats: Malik for his maintainer work on open-trail-report during paid hours, two editors who wanted help with data checks and spelling consistency under the lawyer's guidance from lesson 10, and the partner publisher's liaison developer who was joining Tidewater in September. Rosa projected the cost.

Python Example · A projection of the cost of eight seats, scaling the pilot's usage.
seats_now, seats_next = 4, 8
price = 19                 # US dollars per Business seat per month
pool_per_seat = 1900       # AI credits added to the pool per seat
expected_credits = 8900 * seats_next / seats_now  # scale pilot use
extra = max(0, expected_credits - pool_per_seat * seats_next)
monthly = seats_next * price + extra / 100       # 1 credit = 1 cent
print("seat cost:", seats_next * price)
print("expected credits:", round(expected_credits))
print("extra usage dollars:", round(extra / 100, 2))
print("monthly total:", round(monthly, 2))
print("annual total:", round(monthly * 12, 2))
Output
seat cost: 152
expected credits: 17800
extra usage dollars: 26.0
monthly total: 178.0
annual total: 2136.0

What this code does

The example starts from the pilot: four seats used 8,900 AI credits in a month. It assumes eight seats would use twice as much, works out how far that goes beyond the pool that eight seats would include, and converts the extra credits to dollars at one cent each. It then adds the seat charges. The output shows 152 dollars for seats plus about 26 dollars of extra usage, roughly 178 US dollars a month or 2,136 a year, which is the figure Rosa put in front of the owners.

The projection assumed the new users would use Copilot as heavily as the developers, which Rosa thought unlikely for the editors. She kept the assumption anyway, because it gave the owners a ceiling rather than a hopeful guess, and proposed raising the extra-usage budget to 30 US dollars with an alert at 20.

Worked problem

Is the time saved worth the cost?

Developers reported about 2 hours saved a week. Suppose only half of that is real, and only the five developers count (the three web developers, Malik and the liaison). At a loaded labor cost of 45 US dollars an hour and 46 working weeks a year, how does the value compare with the annual cost of 2,136 US dollars?

Show solution
  1. Hours actually saved per developer per week: 2 x 0.5 = 1 hour.
  2. Hours per year: 5 developers x 1 x 46 weeks = 230 hours.
  3. Value: 230 x 45 = 10,350 US dollars.
  4. Compared with cost: 10,350 is nearly five times 2,136.

Answer: Even with a cautious discount, the value is several times the cost. The case does not depend on the editors' seats paying for themselves, which is why those were treated as a trial.

Rosa noted for the owners that the projection was a ceiling and that the editors' use would probably be lighter.

Step 4: plan and policies

The recommendation stayed on Copilot Business. Enterprise would have needed GitHub Enterprise Cloud for all members, which lesson 2 had shown was far more expensive for features Tidewater did not need. The policy settings from lesson 9 carried over, with three changes for the larger group.

Policy changes for the expansion.
SettingPilotExpansion
Who can get seatsweb team and OwenTeams web, editors-copilot and maintainers-trail-report
Coding agentOn for trailcheckOn for trailcheck and open-trail-report; off for trail-notes
Copilot code reviewAutomatic on openingUnchanged, plus instructions for open-trail-report
Suggestions matching public codeBlockedBlocked
Content exclusiontrailcheck-admin, secrets, volunteer dataPlus /drafts/partner/** in trail-notes
Preview featuresOffOff, reviewed quarterly
Extra usage budgetAlert 15, stop 20 US dollarsAlert 20, stop 30 US dollars

Seats would be assigned by team rather than by person, using a new editors-copilot team so that editors could be added one at a time as they finished the training. Malik would get his seat through the maintainers-trail-report team. The coding agent stayed off in trail-notes because the guidebook text was written by people, per the lawyer's advice.

Step 5: people and training

The plan put as much weight on people as on settings. Every new user would attend a one-hour introduction before getting a seat, read the team rules page, and pair with a pilot user for their first week. The rules page was short on purpose.

The team rules page, kept in the admin repository and linked from every repository's CONTRIBUTING.md.
# Copilot at Tidewater: team rules

1. Use Copilot through your company seat only. No personal plans for company work.
2. Never put passwords, keys or volunteers' personal data in a prompt.
3. Read before Tab. Keep only code you can explain.
4. Agent work goes through a pull request and a human review, like any other work.
5. Copilot review is a helper, not an approval.
6. Guidebook prose is written by people. Copilot may check spelling and consistency.
7. If Copilot suggests a package you do not know, check it before installing.
8. Tell us what works and what does not at the monthly half-hour.

Step 6: risks and how they are handled

The owners asked what could go wrong. The team answered with a short risk register, each risk with an owner.

The rollout risk register.
RiskLikelihoodResponseOwner
Sensitive data in a promptLowRules, exclusions, sample data, briefing at onboardingRosa
Lower code quality from unreviewed suggestionsMediumRequired human review; track bugs after releasePriya
License problems from matched codeLowPublic code blocked; code references checked in open-trail-reportMalik
Costs above budgetLowBudget alert and stop; monthly bill reviewRosa
Over-reliance by newer staffMediumPairing in the first week; explain-before-keep rulePriya
Product changes affecting policies or pricesHighQuarterly check of GitHub Docs and the changelogDmitri

Step 7: the rollout timeline

A timeline from May to December 2027. May to mid-July: pilot with four seats, ending with a review on July 21. August: settings and training prepared; new teams created; rules page published. September: four new seats added by team after training; Dmitri sits GH-300 in late August. October: first monthly review with metrics, bill and survey. December: quarterly review of policies, preview features, prices and the decision to keep, grow or shrink.
The rollout plan, with reviews built in.
Rollout timeline. Every date after the July review is a plan.
WhenWhatWho
Late JulyOwners approve the planOwners, Rosa
AugustCreate teams, update policies, budgets and exclusions; publish rulesRosa, Priya
Late AugustDmitri sits GH-300Dmitri
Early SeptemberIntroductions for new users; seats added by teamPriya
OctoberFirst monthly review: metrics, bill, survey, incidentsRosa, Dmitri
DecemberQuarterly review: policies, previews, prices, keep or changeOwners, Rosa, Priya

Worked problem

Planning the first monthly review

For the October review the team wants every new user to have at least 3 weeks of use. Seats are added on September 7. What is the earliest date for the review that meets that rule, and what four inputs should it look at?

Show solution
  1. Three weeks after September 7 is September 28.
  2. Choosing the first Wednesday on or after that date, in keeping with the series, gives September 29; the team picked Wednesday, October 6, to include a full month of billing.
  3. Inputs: usage metrics, the month's bill against budget, a short survey, and any incidents or near misses.

Answer: September 28 at the earliest; the team chose October 6 to line up with the bill. It reviews metrics, cost, people's experience and incidents.

Step 8: deciding again

The last step is the one that is easiest to forget: a date to decide again. Copilot changes quickly, as the course has shown repeatedly with renamed features, new agents and new billing. The plan set a quarterly review in December to check that the policies still matched the product, that preview features had been assessed, that prices and allowances were still as assumed, and that the goals were still being met. If any of them failed, the options were to adjust, to shrink back to the developers, or to stop.

What the owners decided

The owners approved the expansion to eight seats from September, with the editors' seats as a three-month trial, the raised budget, and the December review. Grace asked for one addition: a note in the annual report to readers that TrailCheck's software was developed with the help of AI tools under human review, while the guidebooks themselves remained written by people. The team agreed that was the right kind of transparency.

For Dmitri the series ended with a study plan for GH-300 in late August, MSBA-304 next for prompting and team practice, and MSBA-309 for exam review. For the team it ended with something more useful than a certificate: a set of decisions they could explain, a budget they could defend, and rules that protected their data and their readers.

Lessons the team took from the pilot

At the end of the session Priya asked each person for one lesson from the pilot. Dmitri said that good context beat bigger models. Lena said that asking Copilot to explain code had taught her more than accepting it. Rosa said that budgets and a rules page made the tool easy to approve. Owen said that Copilot was most useful for structure and least for facts. Malik said that code references and the public code filter mattered more for open source than for company code. Grace said that writing down where Copilot would not be used had made the rest of the decisions easier. Priya added her own: the hardest part of a Copilot rollout is not technical, it is agreeing what good use looks like and checking that it happens.

Rosa also prepared a one-page summary for the owners that fitted the whole recommendation on a single screen: what was asked, what the pilot showed, what it would cost, what the risks were, and when the decision would be reviewed. She had learned from the plan decision in lesson 2 that a short, honest summary with the uncertainties stated plainly was more persuasive than a long one that sounded certain. The owners read it in five minutes and spent the rest of the meeting on the two questions that mattered to them, the editors' use and the note to readers.

Common mistakes

  • MistakeChoosing success measures after seeing the results.

    InsteadWrite goals and measures before the pilot starts.

  • MistakeExpanding seats without updating policies, exclusions and budgets.

    InsteadReview every setting for the larger group before adding seats.

  • MistakeTreating self-reported time savings as exact.

    InsteadDiscount them and look for supporting measures.

  • MistakeApproving a rollout with no date to decide again.

    InsteadSchedule regular reviews of metrics, cost, policies and product changes.

Summary and key terms

  • A rollout starts with goals and measures written before a pilot.
  • Pilot results combine usage metrics, billing, review tracking and people's experience, read with care about what they prove.
  • Cost projections should give a ceiling: seat charges plus expected extra usage.
  • Policies, exclusions, budgets and seat assignment by team must be updated for a larger group.
  • Training, a short rules page and pairing help new users use Copilot well.
  • A risk register with owners and a scheduled review keep the rollout under control as the product changes.
Pilot
A small, time-limited trial with defined goals and measures.
Risk register
A list of risks with likelihood, response and owner.
Loaded labor cost
The full cost of an hour of work, including benefits and overheads.
Enablement
Training and support that help people use a tool well.

Practice problems

Work each problem on paper first, then reveal the answer to check.

  1. Why should pilot goals be written before the pilot?

    Show answer

    So the team cannot choose flattering measures after seeing the results.

  2. Why assign expansion seats by team?

    Show answer

    People can be added as they finish training, and seats follow team membership.

  3. Name three inputs to a monthly Copilot review.

    Show answer

    Any three of: usage metrics, the bill against budget, a survey of users, incidents or near misses.

Lesson quiz · 5 questions

Score 60% or more to complete this lesson.

Take the quiz