Testing strategy: from unit tests to accessibility and security checks

Testing is how a team earns the right to say a feature is done, and how it keeps that claim true as the code changes. CMPUT-301 covered testing fundamentals, unit testing, integration and system testing and test-driven development. CMPUT-401 covered quality strategy and release readiness, and AI-420 covered testing machine learning systems. In a capstone, testing has an extra job: it is evidence. The supervisor, the review panel and the client will ask how you know the product works, and 'we clicked around and it seemed fine' is not an answer. A short written test strategy, a test suite that runs on every pull request, and a record of manual testing sessions are the answer.

Lin, as quality lead, wrote Team Relay's test strategy in week 5: two pages saying what kinds of test the team would write, what each kind is for, which tools they would use, when tests run, what coverage they aimed for and how manual, accessibility, performance and security testing would happen. The strategy was reviewed at the design review and revised twice during the term. Its value was less in the document than in the conversation it forced: the team agreed early that every story's acceptance criteria would become automated tests, and that agreement shaped how they estimated and built every story afterwards.

The test pyramid

The test pyramid describes a healthy balance. At the base are many unit tests that check small pieces of logic in isolation and run in milliseconds. In the middle are integration tests that check how pieces work together, for example a request handler with a real database. At the top are a few end-to-end tests that drive the real application through a browser along the most important user journeys. The higher a test, the more confidence it gives that the whole system works, and the slower, more brittle and harder to debug it is. Team Relay ended the term with 312 automated tests: about 220 unit tests, 77 integration tests and 15 end-to-end journeys, and the whole suite ran in under four minutes in CI.

Unit tests

Unit tests should be fast, independent and focused on behaviour rather than implementation. Test what a function promises, its outputs and effects for given inputs, not how it achieves them, so that refactoring does not break tests. Name tests after the behaviour they check, such as test_taken_shift_is_conflict, so that a failure explains itself. Cover the normal path and the edge cases: empty inputs, boundaries, permissions, duplicates and failures. Acceptance criteria translate directly into tests, which is one reason lesson 4 insisted on writing them in Given, When, Then form.

# A tiny, framework-free version of RescueRoute's "claim a shift" handler, with unit tests.
# The real code uses FastAPI; the logic and the tests have the same shape.
import io, unittest

class Shift:
    def __init__(self, sid, driver=None, cancelled=False):
        self.id, self.driver, self.cancelled = sid, driver, cancelled

def claim(shifts, shift_id, user):
    """Return (status code, message). The user comes from the session, never the request body."""
    if user is None:
        return 401, "sign in first"
    if user["role"] != "driver":
        return 403, "only drivers can claim shifts"
    shift = shifts.get(shift_id)
    if shift is None or shift.cancelled:
        return 404, "no such shift"
    if shift.driver == user["id"]:
        return 200, "already yours"
    if shift.driver is not None:
        return 409, "already taken"
    shift.driver = user["id"]
    return 200, "claimed"

class ClaimTests(unittest.TestCase):
    def setUp(self):
        self.shifts = {1: Shift(1), 2: Shift(2, driver=7), 3: Shift(3, cancelled=True)}
        self.ruth = {"id": 5, "role": "driver"}
    def test_claim_open_shift(self):
        self.assertEqual(claim(self.shifts, 1, self.ruth), (200, "claimed"))
        self.assertEqual(self.shifts[1].driver, 5)
    def test_taken_shift_is_conflict(self):
        self.assertEqual(claim(self.shifts, 2, self.ruth)[0], 409)
    def test_claiming_twice_is_idempotent(self):
        claim(self.shifts, 1, self.ruth)
        self.assertEqual(claim(self.shifts, 1, self.ruth), (200, "already yours"))
    def test_signed_out_user_rejected(self):
        self.assertEqual(claim(self.shifts, 1, None)[0], 401)
    def test_coordinator_cannot_claim(self):
        self.assertEqual(claim(self.shifts, 1, {"id": 1, "role": "coordinator"})[0], 403)
    def test_cancelled_shift_not_found(self):
        self.assertEqual(claim(self.shifts, 3, self.ruth)[0], 404)

suite = unittest.defaultTestLoader.loadTestsFromTestCase(ClaimTests)
result = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
for name in sorted(t for t in dir(ClaimTests) if t.startswith("test_")):
    print("PASS" if not any(name in str(f[0]) for f in result.failures + result.errors) else "FAIL", name)
print(f"{result.testsRun} tests, {len(result.failures)} failures, {len(result.errors)} errors")

PASS test_cancelled_shift_not_found
PASS test_claim_open_shift
PASS test_claiming_twice_is_idempotent
PASS test_coordinator_cannot_claim
PASS test_signed_out_user_rejected
PASS test_taken_shift_is_conflict
6 tests, 0 failures, 0 errors

Look at what these tests encode beyond the happy path. A driver who claims the same shift twice gets 'already yours', not an error, because phones on weak connections often send a request twice; the operation is idempotent. A coordinator cannot claim a shift, because the role check matters for security. A cancelled shift behaves as if it does not exist. Each of these rules came from a conversation with Grace or from the STRIDE review in lesson 5, and each now has a test that will fail if a future change breaks it. Tests are executable documentation of decisions.

Integration and end-to-end tests

Integration tests check the seams: the application talking to a real PostgreSQL database, the HTTP layer parsing requests and producing responses, the worker calling the email service through a test double. Use a real database in a container for these tests rather than a mock, because many bugs live in SQL, migrations and constraints that a mock would not reproduce. Team Relay's CI started a throwaway PostgreSQL service for each run, applied all migrations from scratch and loaded a small fixture data set, which also tested that the migrations worked on an empty database.

External services are different. Do not send real emails or text messages from tests. Replace the email and SMS providers with fakes that record what would have been sent, and test against those. For the provider's real behaviour, use its sandbox or test mode in a small number of manual or scheduled checks. End-to-end tests drive a real browser with a tool such as Playwright or Cypress through journeys like 'a driver signs in, claims a shift and receives a reminder in the fake mailbox'. Keep them few and stable, and make them wait for conditions rather than for fixed times, because flaky end-to-end tests that fail randomly teach the team to ignore failures.

Property-based tests

Example-based tests check the cases you thought of. Property-based tests check rules that must hold for every input, by generating hundreds or thousands of random inputs. For RescueRoute's assignment helper, which suggests drivers for unfilled shifts, the properties were: nobody is assigned a shift they are not available for; nobody exceeds three shifts a week; and a shift is left unfilled only if every available driver is already at the limit. A library such as Hypothesis generates inputs and, when it finds a failure, shrinks it to the smallest failing example. The idea can be shown with the standard library alone.

# Property-based testing by hand: generate many random schedules and check rules that
# must ALWAYS hold, rather than a few hand-picked examples. (Libraries such as
# Hypothesis do this with automatic shrinking of failing cases.)
import random
random.seed(499)

def assign(shifts, drivers, max_per_week=3):
    """Greedy weekly assignment: each shift to the available driver with the fewest shifts so far."""
    load = {d: 0 for d in drivers}
    plan = {}
    for s in sorted(shifts, key=lambda s: s["start"]):
        options = [d for d in s["available"] if load[d] < max_per_week]
        if options:
            d = min(options, key=lambda d: (load[d], d))
            plan[s["id"]] = d
            load[d] += 1
    return plan, load

failures = 0
for trial in range(2000):
    drivers = [f"d{i}" for i in range(random.randint(1, 12))]
    shifts = [{"id": i, "start": random.randint(0, 167),
               "available": random.sample(drivers, random.randint(0, len(drivers)))}
              for i in range(random.randint(0, 30))]
    plan, load = assign(shifts, drivers)
    by_id = {s["id"]: s for s in shifts}
    # Property 1: nobody is assigned a shift they are not available for.
    ok1 = all(d in by_id[sid]["available"] for sid, d in plan.items())
    # Property 2: nobody exceeds the weekly limit.
    ok2 = all(n <= 3 for n in load.values())
    # Property 3: a shift is left unfilled only if every available driver is at the limit.
    ok3 = all(sid in plan or all(load[d] >= 3 for d in s["available"]) for sid, s in by_id.items())
    if not (ok1 and ok2 and ok3):
        failures += 1
print(f"2000 random schedules checked, {failures} property violations")

2000 random schedules checked, 0 property violations

The first version of the assignment helper failed property 1 on several dozen of the random schedules. When a shift had nobody available, or everyone available was already at the limit, it fell back to the least-loaded driver overall, a 'helpful' default someone had added during sprint 2 so that no shift would show as unfilled. The result was drivers being booked for times they had said they could not do, exactly what Grace's private list of reliable drivers existed to avoid. No hand-written example had caught it, because the examples always had someone available. The fixed version shown leaves such shifts unfilled for the coordinator, and passes. Property-based tests are especially good at finding such interactions in scheduling, parsing, serialization and anything with rules that must always hold.

Coverage and its limits

Code coverage measures which lines or branches of code the tests executed. It is useful for finding untested areas, and a sharp drop in a pull request is a warning sign. It is a poor target. A test suite can execute every line and still miss important cases, because coverage counts execution, not checking. Team Relay set a floor of 80 per cent line coverage for the application package, enforced in CI, and treated anything above as a by-product of testing behaviour rather than a goal. The example below reaches full line coverage of a reminder function and still misses a serious bug.

# Line coverage measured with the standard library's tracing hook on a small function.
# 100% line coverage does not mean the code is correct: see the missed edge case below.
import sys, inspect

def reminder_hours(shift_hour, now_hour, quiet_start=22, quiet_end=7):
    """Hours until a reminder should go out: 18 h before the shift, moved out of quiet hours."""
    send_at = shift_hour - 18
    if send_at < now_hour:
        send_at = now_hour
    hour_of_day = send_at % 24
    if hour_of_day >= quiet_start or hour_of_day < quiet_end:
        send_at += (quiet_end - hour_of_day) % 24
    return send_at - now_hour

lines_hit = set()
code = reminder_hours.__code__
def tracer(frame, event, arg):
    if frame.f_code is code and event == "line":
        lines_hit.add(frame.f_lineno)
    return tracer

tests = [(40, 0), (30, 20), (60, 0)]          # (shift hour, now hour) in hours from Monday 00:00
sys.settrace(tracer)
results = [reminder_hours(s, n) for s, n in tests]
sys.settrace(None)
src, first = inspect.getsourcelines(reminder_hours)
body = [first + i for i, l in enumerate(src) if l.strip() and not l.strip().startswith(("def", '"""'))]
print("results:", results)
print(f"line coverage: {len(lines_hit & set(body))}/{len(body)} = {100 * len(lines_hit & set(body)) / len(body):.0f}%")
# An edge case the tests missed: a shift starting soon after quiet hours begin.
print("shift at hour 25 (Tuesday 01:00), now hour 23:", reminder_hours(25, 23), "hours -> after the shift has started!")

results: [31, 0, 42]
line coverage: 7/7 = 100%
shift at hour 25 (Tuesday 01:00), now hour 23: 8 hours -> after the shift has started!

The function moves reminders out of the quiet hours between 22:00 and 07:00, so drivers are not texted at night. All its lines run in the tests. But for a shift at 01:00 when it is already 23:00, it schedules the reminder for 07:00, after the shift has started. The tests never tried a shift less than 18 hours away that also falls in quiet hours. The fix was a rule that a reminder is never later than two hours before the shift, and a property test that checks it for random times. Coverage told the team the code was exercised; only thinking about behaviour told them it was wrong.

Manual and exploratory testing

Automated tests check what the team expected. Exploratory testing looks for what nobody expected, by using the product with a goal and a time box and following anything odd. Team Relay ran a one-hour 'bug bash' before each client demo, in which every member tried to break features they had not built, on phones and laptops, and logged issues. They also kept a short manual test script for things automation could not easily check, such as how a reminder email looks in common mail apps. Record manual testing sessions with date, tester, scope and findings: the record is evidence for the final report.

Accessibility and performance testing

Accessibility testing combines automated checks and people, as lesson 4 explained. Team Relay ran an automated accessibility checker on every page in the end-to-end suite, which failed the build on errors such as missing labels or insufficient contrast, and did a manual keyboard and screen reader pass on new screens each sprint. Performance testing checked the non-functional requirement of pages under two seconds on a mid-range phone: Lighthouse audits ran in CI with a simulated mid-range device and slow network, and a simple load test before the pilot confirmed the entry-level hosting plan handled ten times the expected traffic. At the co-op's scale, a load test is a sanity check, not an engineering project.

Security testing

Security testing in a capstone is defensive and proportionate. Static analysis tools scan source code for common vulnerability patterns, such as unsafe SQL construction or disabled certificate checks. Dependency scanning, set up in lesson 6, finds known vulnerabilities in libraries. Secret scanning keeps credentials out of the repository. Tests for authorization rules, like the role tests above, prove that each route enforces who may use it. CMPUT-331 and CISSP-117 covered these practices. Do not run attack tools against systems you do not own, including your hosting provider's infrastructure, and do not test against the production system with real data; use staging. If the client wants a professional security assessment, that is a separate engagement by qualified people.

When a test fails

Treat a failing test in main as the team's highest priority, because every minute it stays red, people stop trusting the suite and start merging around it. Fix it or revert the change that broke it. When a bug is found by hand or reported by the client, first write a test that reproduces it and fails, then fix the code until the test passes. This catches the bug for good and documents it. Track flaky tests explicitly and fix or delete them; a test that fails one run in twenty is worse than no test, because it trains people to click 're-run'.

Worked examples