Quality strategy and release readiness

Quality is a product decision, not only an engineering one. How good must the Driftwood app be? A schedule page that shows the wrong sailing can strand someone on an island overnight; a misaligned logo on an old tablet harms nobody. A product team cannot test everything equally, so it needs a quality strategy: an agreed view of which qualities matter most, where the risks are, how much testing each area deserves, how bugs are triaged and when a release is good enough to ship. CMPUT-301 covered testing techniques such as unit, integration and system testing, test design and continuous integration. This lesson takes the product manager's view: deciding where quality effort goes and making the release decision.

What quality means for a product

The international standard ISO/IEC 25010 describes product quality through characteristics such as functional suitability, performance efficiency, compatibility, interaction capability (usability), reliability, security, maintainability, portability and, since the 2023 revision, safety. For a given product, a few of these dominate. For Driftwood, functional correctness of schedule data and reliability during morning peaks dominate; accessibility matters because many travelers are older or have disabilities and because public-facing services face legal accessibility duties; security matters once personal phone numbers are stored for SMS alerts. Naming the top qualities with the client early lets the team spend effort on the right things.

Risk-based testing

Risk-based testing allocates test effort in proportion to risk, usually estimated as likelihood of failure times impact of failure. Likelihood rises with complexity, novelty, change frequency and dependence on external systems; impact rises with how many users are affected and how badly. Driftwood's schedule import, which reads timetable files from the client's office system, is both complex and high impact, so it deserves the most attention. Dark mode is neither. The code below divides a 60-hour test budget across areas in proportion to their risk scores.

# Risk-based test planning: spend test effort where likelihood x impact is highest.
areas = {
    # area:                      (likelihood 1-5, impact 1-5)
    "Schedule data import":       (4, 5),
    "Delay banner":               (3, 5),
    "Capacity alert SMS":         (4, 4),
    "Deckhand count entry":       (3, 4),
    "First-visit guide page":     (2, 2),
    "Dark mode":                  (2, 1),
}
budget_hours = 60
scores = {a: l * i for a, (l, i) in areas.items()}
total = sum(scores.values())
print(f"{'area':26s}{'risk':>5s}{'hours':>7s}")
for a, s in sorted(scores.items(), key=lambda kv: -kv[1]):
    print(f"{a:26s}{s:5d}{budget_hours * s / total:7.1f}")

area                       risk  hours
Schedule data import         20   17.4
Capacity alert SMS           16   13.9
Delay banner                 15   13.0
Deckhand count entry         12   10.4
First-visit guide page        4    3.5
Dark mode                     2    1.7

Risk ratings should be agreed with the client, who knows impact better than the team: the owner pointed out that the delay banner is high impact because storms cause sudden cancellations and people rely on the app to decide whether to drive. The resulting plan also decides which test types to use. High-risk areas get automated checks in the pipeline, exploratory sessions before each release and monitoring in production; low-risk areas may get a quick check only. Proportion, not perfection, is the goal.

Exploratory testing

Scripted tests check what the team already knows to check. Exploratory testing, described by Cem Kaner, James Bach and Elisabeth Hendrickson, is simultaneous learning, test design and test execution: a skilled tester investigates the product, guided by a mission, using results to decide what to try next. It finds the bugs nobody thought to write a test for. Session-based test management, developed by Jonathan and James Bach, makes exploratory testing manageable: work happens in time-boxed sessions, typically 60 to 90 minutes, each guided by a charter and ending with a short debrief.

Elisabeth Hendrickson's book Explore It! gives a simple charter template: Explore [target] with [resources] to discover [information]. Good charters are focused enough to guide the session but open enough to allow discovery. They are typically derived from the risk list, from recent changes, and from what users and support staff report.

A session report records the charter, the tester, the time spent, what was tested, bugs found, questions raised and how much time went to setup versus testing. Over several sessions, these reports show coverage of risky areas and help the PM see where quality is uncertain.

Triage: severity and priority

Every bug report needs a triage decision. Two separate scales are used. Severity describes technical impact: critical (crash, data loss, security breach or no workaround for a core function), major (a core function is wrong or badly impaired), minor (a non-core problem or one with an easy workaround) and trivial (cosmetic). Priority describes business urgency: how soon it must be fixed, often P1 (fix now or block the release) to P4 (fix when convenient). The two are related but not the same, and the difference matters.

A crash in a rarely used admin export is high severity but may be low priority. A misspelled island name on the home page is trivial in severity but high priority because it embarrasses the client and confuses tourists. A screen reader skipping the delay banner might be rated major in severity but affects few users; it should still be high priority because it excludes a group from critical safety information and may breach accessibility obligations. The code encodes one team's triage rule; real triage combines a rule like this with discussion.

# Severity x priority triage: severity is technical impact, priority is business urgency.
SEV = {"critical": 4, "major": 3, "minor": 2, "trivial": 1}
def priority(sev, users_affected_share, workaround, near_release):
    score = SEV[sev] + (2 if users_affected_share > 0.25 else 0) - (1 if workaround else 0)
    if near_release:
        score += 1
    return "P1" if score >= 6 else "P2" if score >= 4 else "P3" if score >= 2 else "P4"
bugs = [
    ("Schedule shows yesterday's sailings after midnight", "major", 0.60, False, True),
    ("Crash when SMS number has spaces",                    "critical", 0.05, True, True),
    ("Logo misaligned on old tablets",                      "trivial", 0.02, True, False),
    ("Screen reader skips delay banner",                    "major", 0.03, False, True),
    ("Wrong French accent in island name",                  "minor", 0.30, True, False),
]
for title, sev, share, wk, near in bugs:
    print(f"{priority(sev, share, wk, near)}  {sev:8s}  {title}")

P1  major     Schedule shows yesterday's sailings after midnight
P2  critical  Crash when SMS number has spaces
P4  trivial   Logo misaligned on old tablets
P2  major     Screen reader skips delay banner
P3  minor     Wrong French accent in island name

The rule assigns the screen reader bug only P2 because few users are affected. This is exactly where a rule needs human override: Team Lumen's triage meeting raised it to P1 because of its accessibility impact. Encoding a rule is useful for consistency, but the triage meeting, a short regular session where the PM, a developer and a tester review new bugs, exists for these judgment calls. Triage outcomes are fix now, fix in this release, schedule for later, or will not fix with a recorded reason.

Measuring quality

Quality metrics help the team see trends. Defect escape rate (also called defect leakage) is the share of defects found by users after release, out of all defects found for that release; a falling rate means testing is catching more before release. Other useful measures include the number of open bugs by severity, mean time to fix critical bugs, crash-free session rate, and accessibility issues found per release. DORA metrics, covered in CMPUT-301, include change failure rate, which complements the escape rate from a delivery perspective.

# Defect escape rate per release: bugs found by users / all bugs found for that release.
releases = [
    # release, found before release (testing), found after release (users)
    ("0.1 pilot",   18, 9),
    ("0.2",         25, 7),
    ("0.3",         31, 5),
    ("1.0 summer",  40, 4),
]
for name, before, after in releases:
    rate = after / (before + after)
    bar = "#" * round(rate * 50)
    print(f"{name:11s} escape rate {rate:5.1%} {bar}")

0.1 pilot   escape rate 33.3% #################
0.2         escape rate 21.9% ###########
0.3         escape rate 13.9% #######
1.0 summer  escape rate  9.1% #####

Metrics can be gamed. If testers are rewarded for bugs found, they file trivial duplicates; if developers are rewarded for low bug counts, bugs get relabeled as features or reported informally. Use quality metrics to learn and improve, not to rank people. And interpret escape rate together with usage: a release that few people use will have few escaped defects simply because nobody found them.

User acceptance testing and beta programs

In client work, user acceptance testing (UAT) is the client's confirmation that the release meets agreed acceptance criteria. Good UAT is planned: the PM prepares scenarios based on real tasks, the client's staff run them in a staging environment, and results are recorded with sign-off. For Driftwood, the office manager and two deckhands ran UAT on the delay banner and count entry screen. UAT is not the place to find most bugs, since those should be found earlier, but it confirms fitness for purpose and is often a contractual milestone (lesson 11).

A beta program releases to a group of real users who agree to try early versions and give feedback. Driftwood recruited forty regular commuters as beta testers through the island community association. Beta users find problems that no test environment reproduces, such as rural network dead zones on the drive to the terminal, and they become advocates at launch. Clear expectations, an easy feedback channel and visible responses keep beta testers engaged.

The release readiness decision

Before a release, the team asks: is this good enough to ship? A release readiness checklist makes the decision explicit. Some criteria are hard gates that must pass, such as no open P1 bugs, definition of done met, accessibility checks on changed screens, a tested rollback plan and client sign-off. Others are softer and weighed together. The PM chairs the go or no-go meeting, but the decision should be traceable to evidence. The code below implements a simple version.

# A release readiness check: hard gates must all pass; soft criteria are scored.
gates = {
    "No open P1 bugs": True,
    "Definition of done met for all items": True,
    "Accessibility check (WCAG 2.2 AA) on changed screens": True,
    "Rollback plan tested": False,
    "Client sign-off on acceptance tests": True,
}
soft = {  # criterion: (met, weight)
    "Release notes drafted": (True, 2),
    "Staff briefing held": (True, 3),
    "Monitoring dashboards updated": (False, 3),
    "Support FAQ updated": (True, 1),
    "Open P2 bugs under 5": (True, 2),
}
failed = [g for g, ok in gates.items() if not ok]
score = sum(w for ok, w in soft.values() if ok) / sum(w for _, w in soft.values())
print("hard gates:", "all pass" if not failed else "FAIL -> " + "; ".join(failed))
print(f"soft criteria: {score:.0%} of weighted items met")
print("decision:", "GO" if not failed and score >= 0.8 else "NO-GO")

hard gates: FAIL -> Rollback plan tested
soft criteria: 73% of weighted items met
decision: NO-GO

Here the release is a no-go because the rollback plan has not been tested, even though most other criteria are met. That is the right call: an untested rollback plan is a plan that may not work when it is needed most. The team tests the rollback that afternoon and holds the go decision the next morning. Release readiness is also where the staged rollout plan from lesson 7 is confirmed, so the decision is go to 5 percent, not go to everyone.

Worked examples