MSBA-303 · Lesson 12 of 12 · 20 min
Case study: rolling out Copilot to a team
▶ Video coming soon A video version of this lesson is on the way.
The last session, on July 21, 2027, brought the series together. The four-seat pilot had run for nine weeks, since May 17, and the owners wanted a recommendation: expand Copilot to eight people in September, keep it at four, or stop. Rosa, Priya and Dmitri prepared the recommendation in the Portland boardroom as a working session, using everything from the course. This lesson follows that case study step by step, from goals and pilot results to plan, policies, budget, training and review, and ends with the rollout plan the owners approved. Dates from August onward are plans, not results.
Step 1: goals and success measures
A rollout without goals cannot be judged. Back in May, before the pilot, Priya and Rosa had written three goals and how each would be measured. Writing them first stopped the team from choosing flattering numbers afterwards.
| Goal | Measure | Pilot target |
|---|---|---|
| Developers spend less time on routine code | Self-reported weekly estimate, plus pull request time to merge | A noticeable fall without more defects |
| Code quality does not drop | Bugs found after release; review comments per pull request | No increase |
| Data stays protected | No secrets or personal data in prompts; exclusions working | Zero incidents |
| Cost stays predictable | Monthly bill against budget | Extra usage under 20 US dollars a month |
Step 2: what the pilot showed
Dmitri assembled the results from the usage metrics, the billing pages, the team's review-comment tracking from lesson 6 and a short survey.
| Measure | Before pilot | Pilot (9 weeks) |
|---|---|---|
| Average daily active users | Not applicable | 3.2 of 4 seats |
| Acceptance rate of suggestions | Not applicable | 30 in 100 |
| Median pull request time to merge | 26 hours | 19 hours |
| Bugs found after release, per month | 3 | 3 |
| Copilot review comments applied | Not applicable | 45 in 100 |
| Data incidents | 0 | 0 |
| Extra usage per month | Not applicable | 9 to 13 US dollars |
| Self-reported time saved per developer per week | Not applicable | About 2 hours |
The team was careful about what the results proved. Faster merges came alongside Copilot code review catching simple problems, but the pilot months were also quieter than spring. Bugs did not rise, which met the quality goal. The time saved was self-reported, which tends to overstate. Priya summarized it honestly: "Useful, safe and affordable at our size, with some of the benefit uncertain."
Step 3: the case for expansion
Four more people had asked for seats: Malik for his maintainer work on open-trail-report during paid hours, two editors who wanted help with data checks and spelling consistency under the lawyer's guidance from lesson 10, and the partner publisher's liaison developer who was joining Tidewater in September. Rosa projected the cost.
seats_now, seats_next = 4, 8
price = 19 # US dollars per Business seat per month
pool_per_seat = 1900 # AI credits added to the pool per seat
expected_credits = 8900 * seats_next / seats_now # scale pilot use
extra = max(0, expected_credits - pool_per_seat * seats_next)
monthly = seats_next * price + extra / 100 # 1 credit = 1 cent
print("seat cost:", seats_next * price)
print("expected credits:", round(expected_credits))
print("extra usage dollars:", round(extra / 100, 2))
print("monthly total:", round(monthly, 2))
print("annual total:", round(monthly * 12, 2))seat cost: 152 expected credits: 17800 extra usage dollars: 26.0 monthly total: 178.0 annual total: 2136.0
What this code does
The example starts from the pilot: four seats used 8,900 AI credits in a month. It assumes eight seats would use twice as much, works out how far that goes beyond the pool that eight seats would include, and converts the extra credits to dollars at one cent each. It then adds the seat charges. The output shows 152 dollars for seats plus about 26 dollars of extra usage, roughly 178 US dollars a month or 2,136 a year, which is the figure Rosa put in front of the owners.
The projection assumed the new users would use Copilot as heavily as the developers, which Rosa thought unlikely for the editors. She kept the assumption anyway, because it gave the owners a ceiling rather than a hopeful guess, and proposed raising the extra-usage budget to 30 US dollars with an alert at 20.
Worked problem
Is the time saved worth the cost?
Developers reported about 2 hours saved a week. Suppose only half of that is real, and only the five developers count (the three web developers, Malik and the liaison). At a loaded labor cost of 45 US dollars an hour and 46 working weeks a year, how does the value compare with the annual cost of 2,136 US dollars?
Show solution
- Hours actually saved per developer per week: 2 x 0.5 = 1 hour.
- Hours per year: 5 developers x 1 x 46 weeks = 230 hours.
- Value: 230 x 45 = 10,350 US dollars.
- Compared with cost: 10,350 is nearly five times 2,136.
Answer: Even with a cautious discount, the value is several times the cost. The case does not depend on the editors' seats paying for themselves, which is why those were treated as a trial.
Rosa noted for the owners that the projection was a ceiling and that the editors' use would probably be lighter.
Step 4: plan and policies
The recommendation stayed on Copilot Business. Enterprise would have needed GitHub Enterprise Cloud for all members, which lesson 2 had shown was far more expensive for features Tidewater did not need. The policy settings from lesson 9 carried over, with three changes for the larger group.
| Setting | Pilot | Expansion |
|---|---|---|
| Who can get seats | web team and Owen | Teams web, editors-copilot and maintainers-trail-report |
| Coding agent | On for trailcheck | On for trailcheck and open-trail-report; off for trail-notes |
| Copilot code review | Automatic on opening | Unchanged, plus instructions for open-trail-report |
| Suggestions matching public code | Blocked | Blocked |
| Content exclusion | trailcheck-admin, secrets, volunteer data | Plus /drafts/partner/** in trail-notes |
| Preview features | Off | Off, reviewed quarterly |
| Extra usage budget | Alert 15, stop 20 US dollars | Alert 20, stop 30 US dollars |
Seats would be assigned by team rather than by person, using a new editors-copilot team so that editors could be added one at a time as they finished the training. Malik would get his seat through the maintainers-trail-report team. The coding agent stayed off in trail-notes because the guidebook text was written by people, per the lawyer's advice.
Step 5: people and training
The plan put as much weight on people as on settings. Every new user would attend a one-hour introduction before getting a seat, read the team rules page, and pair with a pilot user for their first week. The rules page was short on purpose.
# Copilot at Tidewater: team rules
1. Use Copilot through your company seat only. No personal plans for company work.
2. Never put passwords, keys or volunteers' personal data in a prompt.
3. Read before Tab. Keep only code you can explain.
4. Agent work goes through a pull request and a human review, like any other work.
5. Copilot review is a helper, not an approval.
6. Guidebook prose is written by people. Copilot may check spelling and consistency.
7. If Copilot suggests a package you do not know, check it before installing.
8. Tell us what works and what does not at the monthly half-hour.Step 6: risks and how they are handled
The owners asked what could go wrong. The team answered with a short risk register, each risk with an owner.
| Risk | Likelihood | Response | Owner |
|---|---|---|---|
| Sensitive data in a prompt | Low | Rules, exclusions, sample data, briefing at onboarding | Rosa |
| Lower code quality from unreviewed suggestions | Medium | Required human review; track bugs after release | Priya |
| License problems from matched code | Low | Public code blocked; code references checked in open-trail-report | Malik |
| Costs above budget | Low | Budget alert and stop; monthly bill review | Rosa |
| Over-reliance by newer staff | Medium | Pairing in the first week; explain-before-keep rule | Priya |
| Product changes affecting policies or prices | High | Quarterly check of GitHub Docs and the changelog | Dmitri |
Step 7: the rollout timeline
| When | What | Who |
|---|---|---|
| Late July | Owners approve the plan | Owners, Rosa |
| August | Create teams, update policies, budgets and exclusions; publish rules | Rosa, Priya |
| Late August | Dmitri sits GH-300 | Dmitri |
| Early September | Introductions for new users; seats added by team | Priya |
| October | First monthly review: metrics, bill, survey, incidents | Rosa, Dmitri |
| December | Quarterly review: policies, previews, prices, keep or change | Owners, Rosa, Priya |
Worked problem
Planning the first monthly review
For the October review the team wants every new user to have at least 3 weeks of use. Seats are added on September 7. What is the earliest date for the review that meets that rule, and what four inputs should it look at?
Show solution
- Three weeks after September 7 is September 28.
- Choosing the first Wednesday on or after that date, in keeping with the series, gives September 29; the team picked Wednesday, October 6, to include a full month of billing.
- Inputs: usage metrics, the month's bill against budget, a short survey, and any incidents or near misses.
Answer: September 28 at the earliest; the team chose October 6 to line up with the bill. It reviews metrics, cost, people's experience and incidents.
Step 8: deciding again
The last step is the one that is easiest to forget: a date to decide again. Copilot changes quickly, as the course has shown repeatedly with renamed features, new agents and new billing. The plan set a quarterly review in December to check that the policies still matched the product, that preview features had been assessed, that prices and allowances were still as assumed, and that the goals were still being met. If any of them failed, the options were to adjust, to shrink back to the developers, or to stop.
What the owners decided
The owners approved the expansion to eight seats from September, with the editors' seats as a three-month trial, the raised budget, and the December review. Grace asked for one addition: a note in the annual report to readers that TrailCheck's software was developed with the help of AI tools under human review, while the guidebooks themselves remained written by people. The team agreed that was the right kind of transparency.
For Dmitri the series ended with a study plan for GH-300 in late August, MSBA-304 next for prompting and team practice, and MSBA-309 for exam review. For the team it ended with something more useful than a certificate: a set of decisions they could explain, a budget they could defend, and rules that protected their data and their readers.
Lessons the team took from the pilot
At the end of the session Priya asked each person for one lesson from the pilot. Dmitri said that good context beat bigger models. Lena said that asking Copilot to explain code had taught her more than accepting it. Rosa said that budgets and a rules page made the tool easy to approve. Owen said that Copilot was most useful for structure and least for facts. Malik said that code references and the public code filter mattered more for open source than for company code. Grace said that writing down where Copilot would not be used had made the rest of the decisions easier. Priya added her own: the hardest part of a Copilot rollout is not technical, it is agreeing what good use looks like and checking that it happens.
Rosa also prepared a one-page summary for the owners that fitted the whole recommendation on a single screen: what was asked, what the pilot showed, what it would cost, what the risks were, and when the decision would be reviewed. She had learned from the plan decision in lesson 2 that a short, honest summary with the uncertainties stated plainly was more persuasive than a long one that sounded certain. The owners read it in five minutes and spent the rest of the meeting on the two questions that mattered to them, the editors' use and the note to readers.
Common mistakes
MistakeChoosing success measures after seeing the results.
InsteadWrite goals and measures before the pilot starts.
MistakeExpanding seats without updating policies, exclusions and budgets.
InsteadReview every setting for the larger group before adding seats.
MistakeTreating self-reported time savings as exact.
InsteadDiscount them and look for supporting measures.
MistakeApproving a rollout with no date to decide again.
InsteadSchedule regular reviews of metrics, cost, policies and product changes.
Summary and key terms
- A rollout starts with goals and measures written before a pilot.
- Pilot results combine usage metrics, billing, review tracking and people's experience, read with care about what they prove.
- Cost projections should give a ceiling: seat charges plus expected extra usage.
- Policies, exclusions, budgets and seat assignment by team must be updated for a larger group.
- Training, a short rules page and pairing help new users use Copilot well.
- A risk register with owners and a scheduled review keep the rollout under control as the product changes.
- Pilot
- A small, time-limited trial with defined goals and measures.
- Risk register
- A list of risks with likelihood, response and owner.
- Loaded labor cost
- The full cost of an hour of work, including benefits and overheads.
- Enablement
- Training and support that help people use a tool well.
Practice problems
Work each problem on paper first, then reveal the answer to check.
Why should pilot goals be written before the pilot?
Show answer
So the team cannot choose flattering measures after seeing the results.
Why assign expansion seats by team?
Show answer
People can be added as they finish training, and seats follow team membership.
Name three inputs to a monthly Copilot review.
Show answer
Any three of: usage metrics, the bill against budget, a survey of users, incidents or near misses.
Lesson quiz · 5 questions
Score 60% or more to complete this lesson.
Create a free account to save your progress, take quizzes and earn a certificate.