Release and operations: deploying, monitoring and handing over

A capstone does not end when the code works. It ends when the client is using the product, the product keeps working without the team, and someone at the client knows what to do when it breaks. Many student projects that worked perfectly at the final demonstration were dead three months later because the hosting trial expired, a password was lost, or the one person who knew how to deploy graduated. This lesson covers the last stretch: releasing to production, running a pilot, monitoring and responding to incidents, and handing over a system the client can keep. CMPUT-401 covered operating products with SLOs, incidents and technical debt; ITIL-141 covered release and deployment management and service levels; AI-420 covered monitoring in production.

Releasing to production

Lesson 6 set up a pipeline that deploys every merge to staging automatically. Production releases add a deliberate step. Before each release, Team Relay ran a short checklist: all tests green on the release commit, the bug bash done with no open critical issues, the database migration tested on a copy of production data structure in staging, release notes written in plain language for Grace, a rollback plan confirmed, and the client informed of the time. They released on Tuesday mornings, never on Fridays and never right before a demo, so that someone was available if anything went wrong. Most releases took ten minutes and were uneventful, which is the goal.

Database migrations are the riskiest part of most releases, because code can be rolled back in seconds but data changes often cannot. Write migrations that are backward compatible with the previous version of the code where possible: add a column before using it, and remove an old one only in a later release after nothing reads it. Test each migration on staging with realistic data before it reaches production. Take a backup or confirm the managed database's point-in-time recovery before any migration that changes or deletes data. Team Relay's pipeline ran migrations automatically before starting the new code, and refused to continue if a migration failed.

Feature flags make releases safer by separating deployment from release. The code for text-message reminders was deployed to production weeks before it was switched on, and the flag let Grace turn it on for five volunteers first, then everyone, and turn it off instantly if replies started misbehaving. Flags have a cost: each is a branch in the code that must be tested both ways and eventually removed. Keep a list of flags with owners and remove them once a feature is fully on.

The pilot

Team Relay ran a four-week pilot in weeks 10 to 13 with 30 volunteer drivers chosen by Grace to include regular and occasional drivers, older and younger volunteers, and the driver who uses a screen reader. The pilot was the real test of the success metrics from lesson 2. Before it started, the team ran a 30-minute training session for Grace, wrote a one-page guide for drivers, and set up a feedback form and a phone number for problems. During the pilot, they met Grace weekly, triaged feedback into the backlog, and released fixes on their normal Tuesday rhythm.

The pilot's results, reported honestly in the final report, were mixed in a useful way. Missed pickups among pilot drivers fell from about 8 per cent to about 4 per cent, mostly because unconfirmed shifts were now noticed the day before. Seventy-one per cent of pilot shifts were confirmed in the system, above the 60 per cent target. Grace's scheduling time fell, but only to about 11 hours a week, missing the target of under 10, because she still scheduled non-pilot drivers the old way. The number of active drivers stayed level, so the guardrail held. The sample was small and the pilot short, and the report said so.

Observability

Observability means being able to understand what a system is doing from the outside, using the data it produces. Its three classic signals are logs, records of individual events; metrics, numbers aggregated over time, such as request rate, error rate and latency; and traces, records of a single request as it moves through components. A small system needs modest versions of each. RescueRoute wrote structured logs with internal IDs and no personal data (lesson 9), exposed basic metrics through the hosting platform, and used an external uptime monitor that checked the sign-in page every minute and sent alerts to the on-call phone. Full distributed tracing, as CMPUT-450 discussed, would have been more complexity than a three-container system needs.

Alert on symptoms users feel, not on every internal event. A good first set of alerts for a capstone is: the site is down, the error rate is above a threshold for several minutes, and background jobs have stopped running. Every alert should be actionable, meaning the person who receives it can do something, and should link to the relevant runbook section. Alerts that fire often and need no action train people to ignore them. Team Relay started with five alerts, removed two noisy ones in the first week of the pilot, and added one for 'no reminders sent in the last 24 hours', after a silent worker failure in week 11 that no existing alert caught.

SLOs and error budgets in practice

The SLO set in lesson 5 was that 99 per cent of page requests succeed in under two seconds, measured over the month. The error budget is the 1 per cent the SLO allows to fail. If the budget is mostly unspent, the team can release freely; if it is nearly used up, they slow down and prioritize reliability work. During the pilot, Ethan computed the SLO weekly from request logs. The program below reproduces the calculation for the four-week pilot, including a slow query on days 10 and 11 and the one real incident on day 17.

# Measuring the SLO during the 4-week pilot from request logs (synthetic):
# 99% of page requests succeed in under 2 seconds. How much error budget was used?
import random
random.seed(499)
days = 28
good = bad = 0
daily = []
for day in range(days):
    n = random.randint(900, 1400)                 # page requests per day
    slow_rate = 0.004
    err_rate = 0.002
    if day == 16:                                 # the incident: database connections exhausted
        err_rate = 0.11
    if day in (9, 10):                            # a slow query before an index was added
        slow_rate = 0.03
    b = sum(1 for _ in range(n) if random.random() < slow_rate + err_rate)
    good += n - b
    bad += b
    daily.append((day + 1, n, b))
total = good + bad
slo = 0.99
budget = (1 - slo) * total
print(f"requests: {total:,}; good: {good:,}; bad: {bad:,}")
print(f"success ratio: {100 * good / total:.2f}% (target {100 * slo:.0f}%)")
print(f"error budget: {budget:,.0f} bad requests allowed; used {bad:,} = {100 * bad / budget:.0f}%")
worst = sorted(daily, key=lambda d: d[2] / d[1], reverse=True)[:3]
for day, n, b in worst:
    print(f"  day {day:2}: {100 * b / n:4.1f}% bad")

requests: 31,015; good: 30,691; bad: 324
success ratio: 98.96% (target 99%)
error budget: 310 bad requests allowed; used 324 = 104%
  day 17:  9.9% bad
  day 10:  3.4% bad
  day 11:  2.5% bad

The pilot narrowly missed its SLO: 98.96 per cent against a target of 99, with the error budget overspent by a few per cent. The single incident day accounted for over a third of the bad requests, and the two days with a slow query, before an index was added, for a good share of the rest. Without those three days the service would have met the target comfortably. Team Relay reported the miss plainly in the final report, explained its causes, and showed what had changed: a new index, a connection-count alert and a test for provider failures. Their supervisor singled this section out as the most professional part of the report, because it showed the team measuring honestly rather than choosing numbers that looked good.

Incidents

An incident is any unplanned event that harms users or threatens to. Responding well is a skill that can be practised. Detect quickly through alerts. Acknowledge, so others know someone is on it. Communicate early with users and the client in plain language, even before you know the cause. Mitigate first, restoring service by restarting, rolling back or switching off a feature flag, and investigate the root cause after. Then write a short blameless review: what happened, the impact, the timeline, the cause, what went well and what will change. CMPUT-401 covered incident management; a student team needs the same steps in a lighter form.

# The pilot's one real incident, reconstructed from the timeline in the incident log.
from datetime import datetime as dt
t = lambda s: dt.strptime("2027-03-24 " + s, "%Y-%m-%d %H:%M")
timeline = [
    ("08:42", "first errors: sign-in pages return 500 for some drivers"),
    ("08:55", "uptime monitor alert to on-call phone (Ethan)"),
    ("09:01", "acknowledged; status note posted to coordinator: use phone list today"),
    ("09:14", "cause found: worker leaks database connections after SMS retries"),
    ("09:20", "mitigation: worker restarted; errors stop"),
    ("09:48", "fix deployed: connections closed in a finally block; test added"),
    ("10:30", "verified over 40 minutes; incident closed; review scheduled"),
]
start, detect, ack, mitigate, close = (t(timeline[i][0]) for i in (0, 1, 2, 4, 6))
for time, what in timeline:
    print(time, what)
mins = lambda a, b: int((b - a).total_seconds() // 60)
print(f"time to detect: {mins(start, detect)} min; to acknowledge: {mins(detect, ack)} min;"
      f" to mitigate: {mins(start, mitigate)} min; to close: {mins(start, close)} min")

08:42 first errors: sign-in pages return 500 for some drivers
08:55 uptime monitor alert to on-call phone (Ethan)
09:01 acknowledged; status note posted to coordinator: use phone list today
09:14 cause found: worker leaks database connections after SMS retries
09:20 mitigation: worker restarted; errors stop
09:48 fix deployed: connections closed in a finally block; test added
10:30 verified over 40 minutes; incident closed; review scheduled
time to detect: 13 min; to acknowledge: 6 min; to mitigate: 38 min; to close: 108 min

The incident on day 17 began when the background worker, retrying text messages after the provider briefly failed, opened new database connections without closing old ones until the database's connection limit was reached and the web application could not connect. Restarting the worker restored service in under 40 minutes; the fix, closing connections in a finally block, and a test that simulates provider failures were deployed half an hour later. The review found two improvements: an alert on database connection count, and a line in the runbook saying a worker restart is always safe. Grace's comment was that the team's status message, 'use the phone list today, we are on it', mattered more to her than how fast they fixed it.

On-call for a student team

Someone must be reachable when the product is in real use, but students have exams and lives. Team Relay used a light on-call rota during the pilot: one person per week was the first contact for alerts between 7:00 and 22:00, with a second person as backup, and no expectation of overnight response because the co-op does not schedule overnight pickups. The rota was agreed with the supervisor and the client in advance, and the runbook made sure any member could handle common problems. After handover, the co-op's own volunteer took over, with a much simpler expectation: check the site is up in the morning and follow the runbook.

# docs/runbook.md (excerpt): what to do when RescueRoute is down

1. Check the status page of the hosting provider. If they report an outage, post
   in the coordinators' chat: "RescueRoute is down; use the phone list today."
2. Open the dashboard -> Logs. Search for ERROR in the last 30 minutes.
3. If errors mention "too many connections": restart the worker service
   (Dashboard -> worker -> Restart). This is safe; reminders resume within 5 min.
4. If the last deploy was within 2 hours and errors started after it:
   roll back (Dashboard -> web -> Deploys -> previous -> Redeploy).
5. If the database is unreachable or data looks wrong: do NOT run migrations by hand.
   Follow "Restore from backup" below and contact the on-call developer.
6. When fixed, write a short incident note: what happened, impact, cause, actions.
Contacts: on-call developer (rota in docs/oncall.md); co-op admin: Grace Delorme.

Backups and recovery

A backup you have never restored is a hope, not a backup. The managed database took daily backups and offered point-in-time recovery for seven days on the co-op's plan. In week 12, Ethan ran a restore drill: he restored the previous night's backup into a new database, pointed a copy of the application at it, checked that recent pickups were present and timed the whole process at 34 minutes. He then wrote the steps into the runbook, with screenshots, so that a volunteer with no database experience could follow them. The drill found one problem: restoring created a database with a new address, which had to be updated in the application settings, a step nobody had written down.

Handing over

Handover is a deliverable with a checklist, not a goodbye email. The client needs four things: access, documentation, knowledge and a plan. Access means every account the system depends on, including hosting, database, domain name, email and SMS providers and the code repository, is owned by the client's organization, with at least two people at the client able to sign in, and student accounts removed. Documentation means a user guide for drivers, an admin guide for coordinators, a technical README, the runbook, the architecture documents and ADRs, and the data inventory. Knowledge means training sessions and a recorded walkthrough. A plan means a maintenance and sustainability plan, discussed below.

Test the documentation on someone who has not seen the project. In week 12, Team Relay asked a volunteer developer from the co-op's network, a web developer at a local company who had agreed to be the co-op's technical contact, to set up the project on his own laptop from the README alone and deploy a trivial change to staging, while the team watched without helping. It took him 50 minutes and he hit three problems: a missing environment variable in the example file, an outdated command, and an unexplained step in the deployment section. All three were fixed the same day. This test is the single best predictor of whether a project survives handover.

Maintenance and sustainability

Software needs care after release: dependency updates for security fixes, small changes as the client's needs shift, and occasional incidents. Write a maintenance plan that is honest about what the client can do. Team Relay's plan named the volunteer developer as technical contact for up to two hours a month; turned on automated dependency update pull requests, which he could merge after the tests passed; listed the monthly costs and when the domain name renews; ranked known issues and future ideas, including shift swaps, from the backlog; and suggested that the co-op offer the project to a future capstone team for the next round of features. A plan like this turns a student project into a maintained product.

Worked examples